Nov 04 00:59:43.309146 ip-10-0-136-96 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 04 00:59:43.309159 ip-10-0-136-96 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 04 00:59:43.309168 ip-10-0-136-96 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 04 00:59:43.309483 ip-10-0-136-96 systemd[1]: Failed to start Kubernetes Kubelet. Nov 04 00:59:53.558838 ip-10-0-136-96 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 04 00:59:53.558853 ip-10-0-136-96 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1b83d85a840c431ca21a7a61c477ca4c -- Nov 04 01:02:05.666648 ip-10-0-136-96 systemd[1]: Starting Kubernetes Kubelet... Nov 04 01:02:06.041382 ip-10-0-136-96 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:02:06.041382 ip-10-0-136-96 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 01:02:06.041382 ip-10-0-136-96 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:02:06.041382 ip-10-0-136-96 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:02:06.046213 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046127 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.046213 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046208 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.046213 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046214 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.046213 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046218 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046222 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046226 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046230 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046233 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046237 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046241 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046244 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046251 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046258 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046263 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046267 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046272 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046276 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046280 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046285 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046289 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046294 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046305 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.046485 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046309 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046314 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046318 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046322 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046326 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046347 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046352 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046356 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046361 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046365 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046369 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046374 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046378 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046382 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046387 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046392 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046396 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046400 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046405 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046410 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.047240 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046414 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046419 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046423 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046428 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046433 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046437 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046441 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046445 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046449 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046453 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046457 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046462 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046466 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046470 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046474 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046478 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046482 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046488 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046492 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046496 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.048081 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046500 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046504 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046508 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046512 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046516 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046521 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046526 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046531 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046535 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046540 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046545 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046549 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046553 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046558 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046565 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046571 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046575 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046580 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046584 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.048941 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046588 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046592 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046598 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046602 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046606 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046612 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046616 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046620 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046625 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046629 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046633 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046637 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046641 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046646 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046650 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046654 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046660 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046664 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046668 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046672 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.049759 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046677 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046681 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046685 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046690 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.046694 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.047963 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.047973 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.047978 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.047982 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.047987 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.047992 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.047996 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048000 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048004 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048009 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048013 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048017 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048022 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048026 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048030 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.050603 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048034 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048038 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048042 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048046 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048051 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048058 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048065 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048070 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048075 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048079 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048083 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048087 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048091 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048095 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048099 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048104 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048108 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048113 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048119 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.051552 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048123 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048128 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048132 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048136 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048141 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048145 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048149 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048153 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048157 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048161 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048166 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048170 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048174 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048178 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048183 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048187 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048191 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048195 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048199 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.052420 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048203 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048208 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048212 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048216 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048222 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048227 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048232 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048237 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048241 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048245 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048250 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048254 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048259 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048264 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048269 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048273 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048278 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048282 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048286 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.052902 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048290 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048295 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048299 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048303 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048307 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048311 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048316 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048321 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048326 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048347 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048364 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048368 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048372 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048377 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048384 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048388 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048393 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048396 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048401 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.053572 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048405 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048409 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048414 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048418 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048423 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048426 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048431 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048436 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048441 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048446 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048450 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048457 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048461 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048465 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.048469 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049359 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049369 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049375 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049380 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049385 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.054027 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049389 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049393 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049398 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049402 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049407 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049411 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049416 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049421 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049425 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049429 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049433 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049437 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049442 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049446 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049453 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049457 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049461 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049465 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049470 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.054567 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049475 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049479 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049483 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049488 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049491 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049496 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049500 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049504 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049509 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049513 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049518 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049522 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049526 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049530 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049536 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049540 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049544 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049548 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049552 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049556 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.055082 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049561 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049565 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049570 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049574 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049578 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049582 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049586 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049592 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049596 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049601 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049605 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049609 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049613 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049618 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049623 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049627 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049631 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049635 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049640 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049644 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.055628 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049648 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049652 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049656 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049661 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049666 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049670 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049674 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049678 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049682 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049687 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049693 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049698 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049703 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049707 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049712 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049716 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049720 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049724 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049729 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.056087 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049734 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049741 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049748 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049753 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049758 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049763 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049767 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049772 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049776 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049780 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049784 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049789 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049793 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049797 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049802 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049806 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049811 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049815 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049820 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.056601 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049824 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049828 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049832 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.049836 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.049968 2583 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.049979 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.049988 2583 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.049995 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050007 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050011 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050018 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050024 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050030 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050037 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050043 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050048 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050053 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050058 2583 flags.go:64] FLAG: --cgroup-root="" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050063 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050068 2583 flags.go:64] FLAG: --client-ca-file="" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050073 2583 flags.go:64] FLAG: --cloud-provider="external" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050078 2583 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050087 2583 flags.go:64] FLAG: --cluster-domain="" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050091 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 01:02:06.057048 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050097 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050101 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050107 2583 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050113 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050118 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050125 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050130 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050134 2583 flags.go:64] FLAG: --contention-profiling="false" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050140 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050144 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050150 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050154 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050161 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050166 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050171 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050176 2583 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050180 2583 flags.go:64] FLAG: --enable-server="true" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050185 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050192 2583 flags.go:64] FLAG: --event-burst="100" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050197 2583 flags.go:64] FLAG: --event-qps="50" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050202 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050209 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050214 2583 flags.go:64] FLAG: --eviction-hard="" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050220 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 01:02:06.057657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050225 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050230 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050236 2583 flags.go:64] FLAG: --eviction-soft="" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050240 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050245 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050250 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050260 2583 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050266 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050270 2583 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050275 2583 flags.go:64] FLAG: --feature-gates="" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050282 2583 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050287 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050292 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050300 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050305 2583 flags.go:64] FLAG: --healthz-port="10248" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050310 2583 flags.go:64] FLAG: --help="false" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050315 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050321 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050325 2583 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050346 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050352 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050358 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050363 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 01:02:06.058245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050368 2583 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050372 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050377 2583 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050382 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050389 2583 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050393 2583 flags.go:64] FLAG: --kube-reserved="" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050400 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050404 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050409 2583 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050414 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050419 2583 flags.go:64] FLAG: --lock-file="" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050424 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050429 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050435 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050443 2583 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050448 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050454 2583 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050459 2583 flags.go:64] FLAG: --logging-format="text" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050464 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050469 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050474 2583 flags.go:64] FLAG: --manifest-url="" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050479 2583 flags.go:64] FLAG: --manifest-url-header="" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050488 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050493 2583 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050499 2583 flags.go:64] FLAG: --max-pods="110" Nov 04 01:02:06.058817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050504 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050509 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050514 2583 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050518 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050523 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050528 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050533 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050544 2583 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050548 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050553 2583 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050558 2583 flags.go:64] FLAG: --pod-cidr="" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050563 2583 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050567 2583 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050574 2583 flags.go:64] FLAG: --pods-per-core="0" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050579 2583 flags.go:64] FLAG: --port="10250" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050584 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050588 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04de64db023bb9d02" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050594 2583 flags.go:64] FLAG: --qos-reserved="" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050599 2583 flags.go:64] FLAG: --read-only-port="10255" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050603 2583 flags.go:64] FLAG: --register-node="true" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050608 2583 flags.go:64] FLAG: --register-with-taints="" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050614 2583 flags.go:64] FLAG: --registry-burst="10" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050619 2583 flags.go:64] FLAG: --registry-qps="5" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050623 2583 flags.go:64] FLAG: --reserved-cpus="" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050629 2583 flags.go:64] FLAG: --reserved-memory="" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050636 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 01:02:06.059472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050641 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050646 2583 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050651 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050658 2583 flags.go:64] FLAG: --runonce="false" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050662 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050667 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050673 2583 flags.go:64] FLAG: --seccomp-default="false" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050677 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050682 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050687 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050692 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050697 2583 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050702 2583 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050706 2583 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050710 2583 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050715 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050720 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050725 2583 flags.go:64] FLAG: --system-cgroups="" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050730 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050739 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050744 2583 flags.go:64] FLAG: --tls-cert-file="" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050749 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050756 2583 flags.go:64] FLAG: --tls-min-version="" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050761 2583 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050765 2583 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 01:02:06.060114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050770 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050775 2583 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050780 2583 flags.go:64] FLAG: --v="2" Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050787 2583 flags.go:64] FLAG: --version="false" Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050794 2583 flags.go:64] FLAG: --vmodule="" Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050800 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.050807 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051027 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051035 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051039 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051046 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051051 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051055 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051060 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051064 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051068 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051072 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051077 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051082 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051087 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051091 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.060715 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051095 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051099 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051103 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051108 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051112 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051116 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051121 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051125 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051130 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051135 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051139 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051143 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051147 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051152 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051156 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051161 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051165 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051169 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051175 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051179 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.061215 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051183 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051189 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051194 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051199 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051203 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051207 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051212 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051216 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051221 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051225 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051229 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051233 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051238 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051242 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051247 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051251 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051256 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051260 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051264 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.061699 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051269 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051273 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051277 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051281 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051285 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051290 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051294 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051298 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051303 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051307 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051311 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051316 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051320 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051325 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051347 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051352 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051357 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051361 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051366 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051370 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.062151 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051374 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051379 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051383 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051387 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051391 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051396 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051400 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051405 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051409 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051414 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051418 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051422 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051426 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051430 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051435 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051439 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051445 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051451 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051455 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.062643 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051460 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051464 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051468 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051472 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051479 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051487 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051493 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051500 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051507 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051513 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051518 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051523 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051529 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051533 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.051573 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:02:06.063096 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051632 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051640 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051644 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051649 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051653 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051657 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051661 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051665 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051670 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051674 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051679 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051683 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051687 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051691 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051696 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051701 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051705 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051710 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051714 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.063502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051719 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051724 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051728 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051732 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051737 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051742 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051747 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051752 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051756 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051761 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051766 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051770 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051774 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051778 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051783 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051787 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051791 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051796 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051800 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051806 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.063945 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051810 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051814 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051818 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051822 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051827 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051831 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051835 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051839 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051845 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051849 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051854 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051858 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051862 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051866 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051870 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051875 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051879 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051883 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051890 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051894 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.064430 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051899 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051903 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051907 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051911 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051915 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051919 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051924 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051930 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051934 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051938 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051943 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051947 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051951 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051955 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051959 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051963 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051968 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051972 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051977 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051981 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.064923 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051987 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051991 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.051995 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052000 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052004 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052008 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052013 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052017 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052022 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052026 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052033 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052042 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052046 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052050 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052054 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052061 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052066 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052071 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052075 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.065402 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052081 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052087 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052091 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052096 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052099 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052103 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052107 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.052111 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.059404 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.059418 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.059433 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.059441 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059520 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059526 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059529 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059531 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059534 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059537 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059540 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.065888 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059543 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059546 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059549 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059551 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059554 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059565 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059568 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059571 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059574 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059577 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059579 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059582 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059584 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059587 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059590 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059593 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059595 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059598 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059600 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.066353 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059603 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059608 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059611 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059615 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059617 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059620 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059623 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059625 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059628 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059630 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059633 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059635 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059638 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059641 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059643 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059646 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059648 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059651 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059653 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.066802 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059662 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059664 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059667 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059669 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059672 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059674 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059677 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059680 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059682 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059685 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059688 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059690 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059693 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059696 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059698 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059701 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059704 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059707 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059710 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059712 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.067280 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059715 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059717 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059720 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059722 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059725 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059728 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059730 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059733 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059735 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059738 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059740 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059743 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059745 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059753 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059756 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059758 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059761 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059763 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059766 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059775 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.067831 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059778 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059780 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059783 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059785 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059787 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059791 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059793 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059796 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059799 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059802 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059804 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059808 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059812 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059815 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059817 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059820 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059823 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059825 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059828 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.068291 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059830 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.059833 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.059861 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060044 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060048 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060052 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060055 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060065 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060069 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060073 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060077 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060081 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060085 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060088 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060092 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060096 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.068748 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060100 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060103 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060107 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060112 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060116 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060121 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060124 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060127 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060129 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060132 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060135 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060138 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060140 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060143 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060145 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060148 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060150 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060153 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060156 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.069383 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060158 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060161 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060164 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060166 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060169 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060178 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060181 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060183 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060186 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060189 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060191 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060194 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060196 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060199 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060201 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060204 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060206 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060209 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060211 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060214 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.070052 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060216 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060219 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060221 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060224 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060226 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060229 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060231 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060235 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060239 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060242 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060245 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060248 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060250 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060253 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060255 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060258 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060262 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060265 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060274 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.070565 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060277 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060280 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060282 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060285 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060288 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060290 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060293 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060296 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060298 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060300 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060303 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060306 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060308 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060311 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060313 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060316 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060318 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060321 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060323 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.071019 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060326 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060328 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060342 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060345 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060348 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060351 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060353 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060356 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060358 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060361 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060364 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060366 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060369 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060377 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060379 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060382 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.071502 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.060408 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060454 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060459 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060462 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060466 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060469 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060473 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060476 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060479 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060482 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060484 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060487 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060490 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060492 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060495 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060497 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060499 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060502 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060504 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:06.071870 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060507 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060510 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060512 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060515 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060518 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060520 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060522 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060525 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060528 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060530 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060533 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060541 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060543 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060546 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060548 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060554 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060557 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060559 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060562 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060564 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:06.072321 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060567 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060569 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060572 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060574 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060577 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060580 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060583 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060585 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060588 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060591 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060593 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060596 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060598 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060601 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060604 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060606 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060609 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060611 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060614 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060616 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:06.072864 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060619 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060621 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060623 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060626 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060634 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060636 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060639 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060643 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060645 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060648 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060651 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060653 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060655 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060658 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060660 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060663 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060665 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060668 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060670 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:06.073407 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060673 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060675 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060678 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060680 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060683 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060685 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060688 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060691 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060693 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060696 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060698 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060701 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060703 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060706 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060708 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060711 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060713 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060716 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060730 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060733 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:06.073879 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060736 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060739 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060741 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060744 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060748 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060751 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060754 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060758 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:06.060761 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.060819 2583 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.064367 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.065233 2583 server.go:1014] "Starting client certificate rotation" Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.065346 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.065383 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.066914 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:02:06.074389 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.071350 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:02:06.081111 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.081094 2583 log.go:25] "Validated CRI v1 runtime API" Nov 04 01:02:06.084724 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.084709 2583 log.go:25] "Validated CRI v1 image API" Nov 04 01:02:06.085562 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.085539 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 01:02:06.090528 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.090503 2583 fs.go:135] Filesystem UUIDs: map[38ad2dac-9884-4e02-b618-beda4164003b:/dev/nvme0n1p3 52caf3fb-82b1-48b5-b10b-a67a63c9e575:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 01:02:06.090615 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.090527 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 01:02:06.095618 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.095504 2583 manager.go:217] Machine: {Timestamp:2026-11-04 01:02:06.094204928 +0000 UTC m=+0.322859307 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100128 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db052ba5ee4392b69710381cf99a6 SystemUUID:ec2db052-ba5e-e439-2b69-710381cf99a6 BootID:1b83d85a-840c-431c-a21a-7a61c477ca4c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:92:fa:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:92:fa:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:1d:3c:c6:c0:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 01:02:06.095618 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.095610 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 01:02:06.095772 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.095708 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 01:02:06.097480 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.097452 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 01:02:06.097602 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.097481 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-96.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 01:02:06.097654 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.097603 2583 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 01:02:06.097654 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.097611 2583 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 01:02:06.097654 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.097636 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:02:06.097654 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.097649 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:02:06.098782 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.098771 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 01:02:06.102823 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.102805 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 01:02:06.103548 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.103536 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 01:02:06.103663 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.103654 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 01:02:06.103692 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.103685 2583 kubelet.go:493] "Attempting to sync node with API server" Nov 04 01:02:06.103719 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.103695 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 01:02:06.103719 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.103704 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 01:02:06.103794 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.103724 2583 kubelet.go:405] "Adding apiserver pod source" Nov 04 01:02:06.103794 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.103732 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 01:02:06.104077 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.104056 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:02:06.106033 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.106017 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 01:02:06.112616 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.112600 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:02:06.113314 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113299 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 01:02:06.113410 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113356 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 01:02:06.113561 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113547 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 01:02:06.113619 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113568 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 01:02:06.113619 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113578 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 01:02:06.113619 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113587 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 01:02:06.113619 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113597 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 01:02:06.113619 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113607 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 01:02:06.113930 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113921 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 01:02:06.113979 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113935 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 01:02:06.113979 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113947 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 01:02:06.113979 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113958 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 01:02:06.113979 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113971 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 01:02:06.114117 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.113984 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 01:02:06.114478 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.114462 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 01:02:06.114478 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.114478 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 01:02:06.117534 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.117514 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-96.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 01:02:06.118235 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.118217 2583 server.go:1263] "Started kubelet" Nov 04 01:02:06.118348 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.118302 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 01:02:06.118388 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.118328 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 01:02:06.118428 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.118409 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 01:02:06.118882 ip-10-0-136-96 systemd[1]: Started Kubernetes Kubelet. Nov 04 01:02:06.120211 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.120196 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 01:02:06.121343 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.121313 2583 server.go:317] "Adding debug handlers to kubelet server" Nov 04 01:02:06.124352 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.124326 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 01:02:06.127551 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.127532 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 01:02:06.128681 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.128666 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 01:02:06.128798 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.128777 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.129394 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129376 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 01:02:06.129394 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129379 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 01:02:06.129542 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129406 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 01:02:06.129542 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129463 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4vvzh" Nov 04 01:02:06.129542 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129498 2583 reconstruct.go:97] "Volume reconstruction finished" Nov 04 01:02:06.129542 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129505 2583 reconciler.go:29] "Reconciler: start to sync state" Nov 04 01:02:06.129725 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129663 2583 factory.go:153] Registering CRI-O factory Nov 04 01:02:06.129725 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129674 2583 factory.go:223] Registration of the crio container factory successfully Nov 04 01:02:06.129798 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129754 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 01:02:06.129798 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129761 2583 factory.go:55] Registering systemd factory Nov 04 01:02:06.129798 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129766 2583 factory.go:223] Registration of the systemd container factory successfully Nov 04 01:02:06.129798 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129777 2583 factory.go:103] Registering Raw factory Nov 04 01:02:06.129798 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.129786 2583 manager.go:1199] Started watching for new ooms in manager Nov 04 01:02:06.130567 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.130537 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 01:02:06.130643 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.130625 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-96.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 01:02:06.130746 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.130724 2583 manager.go:319] Starting recovery of all containers Nov 04 01:02:06.131594 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.130650 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.us-east-2.compute.internal.18e4b1dfbdaa6468 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 01:02:06.11818404 +0000 UTC m=+0.346838418,LastTimestamp:2026-11-04 01:02:06.11818404 +0000 UTC m=+0.346838418,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.us-east-2.compute.internal,}" Nov 04 01:02:06.138204 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.138141 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-96.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 01:02:06.141089 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.141064 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4vvzh" Nov 04 01:02:06.142623 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.142606 2583 manager.go:324] Recovery completed Nov 04 01:02:06.147126 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.147087 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 01:02:06.147916 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.147901 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:06.149651 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.149634 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:06.149749 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.149669 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:06.149749 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.149684 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:06.150210 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.150198 2583 cpu_manager.go:226] "Starting" policy="none" Nov 04 01:02:06.150210 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.150208 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 01:02:06.150304 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.150222 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 01:02:06.150304 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.150262 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:06.150950 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.150935 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:06.151020 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.150959 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:06.151020 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.150969 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:06.152169 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.152157 2583 policy_none.go:50] "Start" Nov 04 01:02:06.152207 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.152180 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 01:02:06.152207 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.152190 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 01:02:06.154139 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.154128 2583 policy_none.go:44] "Start" Nov 04 01:02:06.182056 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.182025 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.184716 2583 manager.go:341] "Starting Device Plugin manager" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.184759 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.184771 2583 server.go:87] "Starting device plugin registration server" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.185028 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.185040 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.185242 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.185311 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.185319 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.185850 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.185911 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.199068 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.200305 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.200324 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.200357 2583 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.200390 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 01:02:06.210595 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.203508 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:06.285185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.285168 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:06.285994 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.285979 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:06.286055 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.286010 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:06.286055 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.286025 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:06.286055 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.286053 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.292586 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.292551 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.292586 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.292571 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-96.us-east-2.compute.internal\": node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.300849 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.300832 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal"] Nov 04 01:02:06.300920 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.300887 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:06.304534 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.304519 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:06.304593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.304541 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:06.304593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.304550 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:06.306501 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.306490 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:06.306634 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.306620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.306715 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.306649 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:06.307187 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.307161 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:06.307187 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.307196 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:06.307299 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.307196 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.307299 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.307212 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:06.307299 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.307238 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:06.307299 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.307256 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:06.307299 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.307265 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:06.309287 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.309274 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.309325 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.309298 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:06.309970 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.309957 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:06.310022 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.309980 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:06.310022 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.309990 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:06.346703 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.346686 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" node="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.349359 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.349328 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" node="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.407640 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.407620 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.430983 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.430961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.431052 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.430990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/916a1141c4b97a1387808b2945274410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"916a1141c4b97a1387808b2945274410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.431052 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.431008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.508260 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.508233 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.531586 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.531565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.531647 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.531591 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/916a1141c4b97a1387808b2945274410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"916a1141c4b97a1387808b2945274410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.531647 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.531625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.531709 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.531656 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.531709 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.531665 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.531709 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.531675 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/916a1141c4b97a1387808b2945274410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"916a1141c4b97a1387808b2945274410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.609019 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.608956 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.651458 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.651436 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.653738 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:06.653583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:06.709169 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.709147 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.809694 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.809660 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:06.910173 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:06.910111 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:07.010662 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:07.010631 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:07.065091 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.065065 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 01:02:07.065556 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.065223 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 01:02:07.111311 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:07.111288 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:07.147181 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.147145 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:57:06 +0000 UTC" deadline="2028-03-31 19:50:02.929763274 +0000 UTC" Nov 04 01:02:07.147181 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.147175 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12330h47m55.782591661s" Nov 04 01:02:07.212330 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:07.212274 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:07.268683 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.268659 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:07.312879 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:07.312854 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:07.413342 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:07.413300 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:07.513917 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:07.513846 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:07.551510 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:07.551465 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95ae3556f824717465d708137d91c49.slice/crio-557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e WatchSource:0}: Error finding container 557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e: Status 404 returned error can't find the container with id 557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e Nov 04 01:02:07.553180 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:07.553158 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916a1141c4b97a1387808b2945274410.slice/crio-00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39 WatchSource:0}: Error finding container 00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39: Status 404 returned error can't find the container with id 00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39 Nov 04 01:02:07.558764 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.558750 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:02:07.582879 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.582860 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:07.629622 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.629597 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:07.643508 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.643487 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:02:07.644362 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.644335 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:07.657153 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.657130 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:02:07.688643 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:07.688619 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:08.060696 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.060666 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:08.107425 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.107391 2583 apiserver.go:52] "Watching apiserver" Nov 04 01:02:08.117828 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.117804 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 01:02:08.118121 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.118103 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pndwj","openshift-network-operator/iptables-alerter-2lkql","openshift-multus/multus-4qqzm","openshift-ovn-kubernetes/ovnkube-node-fqgxv","kube-system/global-pull-secret-syncer-h7zqj","kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4zdt9","kube-system/konnectivity-agent-cs7zr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b","openshift-image-registry/node-ca-pzkfb","openshift-network-diagnostics/network-check-target-tpsdw","openshift-dns/node-resolver-9cpl7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-4kzpr"] Nov 04 01:02:08.122113 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.122096 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.124328 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.124304 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.126651 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.126622 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.126778 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.126697 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 01:02:08.126858 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.126813 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n5n45\"" Nov 04 01:02:08.127050 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.127035 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 01:02:08.128454 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.128423 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pvmvv\"" Nov 04 01:02:08.128454 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.128426 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 01:02:08.128652 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.128514 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 01:02:08.128725 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.128711 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 01:02:08.129578 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.129557 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.130148 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.130130 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 01:02:08.130718 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.130599 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 01:02:08.130718 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.130650 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h8pc2\"" Nov 04 01:02:08.130718 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.130694 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 01:02:08.132066 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.132048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.133429 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.133400 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dh9vh\"" Nov 04 01:02:08.133526 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.133456 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 01:02:08.133586 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.133565 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 01:02:08.134597 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.134581 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 01:02:08.134663 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.134597 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 01:02:08.134663 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.134610 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 01:02:08.134663 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.134648 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 01:02:08.135101 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.135083 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:08.135199 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.135163 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:08.137518 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.137497 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.137618 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.137578 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:08.138886 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.138862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-env-overrides\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.138993 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.138892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-script-lib\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.138993 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.138910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58d7d01-e446-4636-8aa6-55302737e478-hosts-file\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.138993 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.138925 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-var-lib-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.138993 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.138948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-ovn\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.138993 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.138984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-node-log\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139000 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58d7d01-e446-4636-8aa6-55302737e478-tmp-dir\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139017 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-registration-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2c7r\" (UniqueName: \"kubernetes.io/projected/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kube-api-access-p2c7r\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139114 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-device-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139186 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-sys-fs\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-log-socket\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139253 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.139277 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139273 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-netns\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139299 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-systemd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc46f559-06ba-4445-8048-f29d73b9f168-agent-certs\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139405 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1492af-3805-4346-8db7-55e5db78f06d-ovn-node-metrics-cert\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-socket-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139485 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-kubelet\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139516 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-slash\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-netd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139564 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-config\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc46f559-06ba-4445-8048-f29d73b9f168-konnectivity-ca\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.139636 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-systemd-units\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.140104 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139670 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-etc-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.140104 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139700 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.140104 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139732 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lrg\" (UniqueName: \"kubernetes.io/projected/b58d7d01-e446-4636-8aa6-55302737e478-kube-api-access-j4lrg\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.140104 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.140104 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139811 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-bin\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.140104 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139850 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqzb\" (UniqueName: \"kubernetes.io/projected/7d1492af-3805-4346-8db7-55e5db78f06d-kube-api-access-sqqzb\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.140104 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.139948 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.144678 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.144608 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.146641 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.146613 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:08.146743 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.146706 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:08.149061 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.149042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.151444 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.151423 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.171416 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.171397 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 01:02:08.171525 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.171462 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dc9b" Nov 04 01:02:08.182187 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.182164 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dc9b" Nov 04 01:02:08.204478 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.204439 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"916a1141c4b97a1387808b2945274410","Type":"ContainerStarted","Data":"00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39"} Nov 04 01:02:08.205433 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.205412 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"a95ae3556f824717465d708137d91c49","Type":"ContainerStarted","Data":"557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e"} Nov 04 01:02:08.230446 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.230424 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 01:02:08.235693 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.235674 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78n7j\"" Nov 04 01:02:08.235828 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.235799 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 01:02:08.235947 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.235866 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cfm9p\"" Nov 04 01:02:08.235947 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.235868 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 01:02:08.235947 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.235885 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 01:02:08.235947 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.235910 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 01:02:08.235947 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.235917 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237234 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237294 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237316 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq94r\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237331 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237346 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237717 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237759 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237763 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237826 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 01:02:08.237906 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237864 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhcr5\"" Nov 04 01:02:08.238444 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237932 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 01:02:08.238444 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.237947 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zw2l8\"" Nov 04 01:02:08.240367 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-socket-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.240482 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240389 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-kubelet\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.240482 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-slash\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.240482 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-config\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.240705 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240495 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc46f559-06ba-4445-8048-f29d73b9f168-konnectivity-ca\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.240705 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysconfig\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.240705 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240594 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-kubelet\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.240705 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240628 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-slash\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.240919 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240726 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-var-lib-kubelet\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.240919 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-systemd-units\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.240919 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240770 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-socket-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.240919 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.240919 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240889 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.241185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240943 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-systemd-units\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.241185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.240990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-kubernetes\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.241185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-netns\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.241185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241072 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-hostroot\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.241185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9z5j\" (UniqueName: \"kubernetes.io/projected/e40442e9-f920-48ca-a3bd-8c19c022d168-kube-api-access-g9z5j\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.241185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.241185 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c61d96e-9340-453b-854a-944a058cac37-host\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241210 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-config\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241231 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-bin\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241189 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc46f559-06ba-4445-8048-f29d73b9f168-konnectivity-ca\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-dbus\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241283 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-bin\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241346 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-conf-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241420 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e40442e9-f920-48ca-a3bd-8c19c022d168-iptables-alerter-script\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.241489 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-script-lib\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241610 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58d7d01-e446-4636-8aa6-55302737e478-hosts-file\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241543 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58d7d01-e446-4636-8aa6-55302737e478-hosts-file\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-k8s-cni-cncf-io\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241703 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn69b\" (UniqueName: \"kubernetes.io/projected/8a9f7134-5311-413e-8a94-4170b61d71dd-kube-api-access-qn69b\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-var-lib-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241817 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-sys\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241852 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-bin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241883 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-script-lib\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-etc-kubernetes\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.241940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2c7r\" (UniqueName: \"kubernetes.io/projected/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kube-api-access-p2c7r\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242038 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-conf\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242063 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-system-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.242086 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242070 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-var-lib-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.242629 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-multus\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.242629 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40442e9-f920-48ca-a3bd-8c19c022d168-host-slash\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.242629 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.242797 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242653 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:08.242797 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242688 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-device-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.242797 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-sys-fs\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.242797 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-log-socket\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.242797 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242790 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-log-socket\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242833 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-lib-modules\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242839 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-device-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242846 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-sys-fs\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-etc-tuned\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4g8g\" (UniqueName: \"kubernetes.io/projected/629a09a7-b384-4b7b-82a0-f91930c909aa-kube-api-access-x4g8g\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-systemd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc46f559-06ba-4445-8048-f29d73b9f168-agent-certs\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.242978 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c61d96e-9340-453b-854a-944a058cac37-serviceca\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243007 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-systemd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243063 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243042 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-netd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243092 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243124 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243138 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-netd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lrg\" (UniqueName: \"kubernetes.io/projected/b58d7d01-e446-4636-8aa6-55302737e478-kube-api-access-j4lrg\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243175 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-cni-binary-copy\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-tmp\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-etc-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243278 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243327 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqzb\" (UniqueName: \"kubernetes.io/projected/7d1492af-3805-4346-8db7-55e5db78f06d-kube-api-access-sqqzb\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243354 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-etc-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243354 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 01:02:08.243473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-systemd\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243500 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ch9l\" (UniqueName: \"kubernetes.io/projected/e63e384d-b749-410c-a347-b35ac0f4a318-kube-api-access-8ch9l\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-system-cni-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243604 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-cnibin\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-env-overrides\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243696 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-modprobe-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243740 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-kubelet-config\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243772 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-ovn\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243798 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-node-log\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243840 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-node-log\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243842 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-ovn\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58d7d01-e446-4636-8aa6-55302737e478-tmp-dir\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243963 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-cnibin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.243989 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-socket-dir-parent\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244018 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-registration-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.244098 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244051 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244099 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-registration-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244109 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-run\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244133 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244175 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-multus-daemon-config\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-host\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244244 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddvm\" (UniqueName: \"kubernetes.io/projected/0b221911-1895-4385-b628-c84f0caefe17-kube-api-access-tddvm\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244285 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-kubelet\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244413 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntf64\" (UniqueName: \"kubernetes.io/projected/4c61d96e-9340-453b-854a-944a058cac37-kube-api-access-ntf64\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244549 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244563 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-env-overrides\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-netns\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244606 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58d7d01-e446-4636-8aa6-55302737e478-tmp-dir\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244640 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-multus-certs\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244602 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-netns\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.244868 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.244670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1492af-3805-4346-8db7-55e5db78f06d-ovn-node-metrics-cert\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.247662 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.247640 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1492af-3805-4346-8db7-55e5db78f06d-ovn-node-metrics-cert\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.247741 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.247724 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc46f559-06ba-4445-8048-f29d73b9f168-agent-certs\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.255446 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.255423 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2c7r\" (UniqueName: \"kubernetes.io/projected/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kube-api-access-p2c7r\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.255601 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.255579 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lrg\" (UniqueName: \"kubernetes.io/projected/b58d7d01-e446-4636-8aa6-55302737e478-kube-api-access-j4lrg\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.256161 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.256145 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqzb\" (UniqueName: \"kubernetes.io/projected/7d1492af-3805-4346-8db7-55e5db78f06d-kube-api-access-sqqzb\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.345945 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.345870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-cni-binary-copy\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.345945 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.345900 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-tmp\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.345945 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.345918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.345945 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.345936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-systemd\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.346051 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346072 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch9l\" (UniqueName: \"kubernetes.io/projected/e63e384d-b749-410c-a347-b35ac0f4a318-kube-api-access-8ch9l\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-system-cni-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.346126 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret podName:389e7a15-7659-450c-9483-d59ea607e94d nodeName:}" failed. No retries permitted until 2026-11-04 01:02:08.846104525 +0000 UTC m=+3.074758915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret") pod "global-pull-secret-syncer-h7zqj" (UID: "389e7a15-7659-450c-9483-d59ea607e94d") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346173 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346173 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-systemd\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346193 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-cnibin\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346210 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-system-cni-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346227 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-modprobe-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346231 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-cnibin\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.346271 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-kubelet-config\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-cnibin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-socket-dir-parent\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346360 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-modprobe-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346380 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-kubelet-config\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-run\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-multus-daemon-config\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346429 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-cnibin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-host\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346475 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddvm\" (UniqueName: \"kubernetes.io/projected/0b221911-1895-4385-b628-c84f0caefe17-kube-api-access-tddvm\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346482 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346491 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-socket-dir-parent\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346490 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-run\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-kubelet\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346537 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-kubelet\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346543 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-host\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.346884 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346570 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf64\" (UniqueName: \"kubernetes.io/projected/4c61d96e-9340-453b-854a-944a058cac37-kube-api-access-ntf64\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-multus-certs\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysconfig\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-var-lib-kubelet\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-kubernetes\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346693 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-multus-certs\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-netns\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346737 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysconfig\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346730 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-netns\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346739 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-hostroot\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346774 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-hostroot\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9z5j\" (UniqueName: \"kubernetes.io/projected/e40442e9-f920-48ca-a3bd-8c19c022d168-kube-api-access-g9z5j\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346801 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-var-lib-kubelet\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346844 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c61d96e-9340-453b-854a-944a058cac37-host\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346856 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-kubernetes\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-dbus\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.347761 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-conf-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e40442e9-f920-48ca-a3bd-8c19c022d168-iptables-alerter-script\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-k8s-cni-cncf-io\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn69b\" (UniqueName: \"kubernetes.io/projected/8a9f7134-5311-413e-8a94-4170b61d71dd-kube-api-access-qn69b\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347031 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-dbus\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347059 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-conf-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347072 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-sys\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-bin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347133 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.346897 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c61d96e-9340-453b-854a-944a058cac37-host\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-etc-kubernetes\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347180 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-conf\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347205 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-system-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-multus\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347256 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-k8s-cni-cncf-io\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347264 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40442e9-f920-48ca-a3bd-8c19c022d168-host-slash\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.348593 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347311 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40442e9-f920-48ca-a3bd-8c19c022d168-host-slash\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347332 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347366 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-lib-modules\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-etc-tuned\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4g8g\" (UniqueName: \"kubernetes.io/projected/629a09a7-b384-4b7b-82a0-f91930c909aa-kube-api-access-x4g8g\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347459 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-multus-daemon-config\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347467 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-sys\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347459 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-cni-binary-copy\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347452 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-conf\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347410 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-bin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.347544 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347553 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-lib-modules\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347549 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-multus\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347581 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c61d96e-9340-453b-854a-944a058cac37-serviceca\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.347593 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs podName:629a09a7-b384-4b7b-82a0-f91930c909aa nodeName:}" failed. No retries permitted until 2026-11-04 01:02:08.847576322 +0000 UTC m=+3.076230707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs") pod "network-metrics-daemon-pndwj" (UID: "629a09a7-b384-4b7b-82a0-f91930c909aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347544 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-system-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.349112 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347699 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-etc-kubernetes\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.349682 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.347783 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e40442e9-f920-48ca-a3bd-8c19c022d168-iptables-alerter-script\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.349682 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.348107 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c61d96e-9340-453b-854a-944a058cac37-serviceca\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.349682 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.348433 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.349682 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.348483 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.349682 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.348498 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.349682 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.348843 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-tmp\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.349682 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.349662 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-etc-tuned\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.363537 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.363507 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddvm\" (UniqueName: \"kubernetes.io/projected/0b221911-1895-4385-b628-c84f0caefe17-kube-api-access-tddvm\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.365518 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.365497 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:02:08.365642 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.365524 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:02:08.365642 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.365536 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bt98j for pod openshift-network-diagnostics/network-check-target-tpsdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:08.365642 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.365591 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j podName:def4f36e-3bef-48de-acea-cc0e75f0616a nodeName:}" failed. No retries permitted until 2026-11-04 01:02:08.865579028 +0000 UTC m=+3.094233394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bt98j" (UniqueName: "kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j") pod "network-check-target-tpsdw" (UID: "def4f36e-3bef-48de-acea-cc0e75f0616a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:08.367697 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.367666 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ch9l\" (UniqueName: \"kubernetes.io/projected/e63e384d-b749-410c-a347-b35ac0f4a318-kube-api-access-8ch9l\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.368001 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.367967 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4g8g\" (UniqueName: \"kubernetes.io/projected/629a09a7-b384-4b7b-82a0-f91930c909aa-kube-api-access-x4g8g\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:08.368289 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.368272 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntf64\" (UniqueName: \"kubernetes.io/projected/4c61d96e-9340-453b-854a-944a058cac37-kube-api-access-ntf64\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.368345 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.368313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn69b\" (UniqueName: \"kubernetes.io/projected/8a9f7134-5311-413e-8a94-4170b61d71dd-kube-api-access-qn69b\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.369842 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.369821 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9z5j\" (UniqueName: \"kubernetes.io/projected/e40442e9-f920-48ca-a3bd-8c19c022d168-kube-api-access-g9z5j\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.434323 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.434281 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:08.442427 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.442401 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:08.443646 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.443617 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc46f559_06ba_4445_8048_f29d73b9f168.slice/crio-6acb85aa87fd466a15e5b0ec37bc65b4ff6c13094b4983dcc3c72da4386ed66b WatchSource:0}: Error finding container 6acb85aa87fd466a15e5b0ec37bc65b4ff6c13094b4983dcc3c72da4386ed66b: Status 404 returned error can't find the container with id 6acb85aa87fd466a15e5b0ec37bc65b4ff6c13094b4983dcc3c72da4386ed66b Nov 04 01:02:08.451159 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.451136 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58d7d01_e446_4636_8aa6_55302737e478.slice/crio-40cde1a64fddbed450b03fcede0207e6a378f9a3feb53648633d4bac03e24545 WatchSource:0}: Error finding container 40cde1a64fddbed450b03fcede0207e6a378f9a3feb53648633d4bac03e24545: Status 404 returned error can't find the container with id 40cde1a64fddbed450b03fcede0207e6a378f9a3feb53648633d4bac03e24545 Nov 04 01:02:08.456629 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.456611 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:08.466431 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.466396 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3b56a9_d22c_436d_bb4d_d492b9a385f4.slice/crio-4df9ed73329a05017dc55b7b9c9f6cb0af932ea3c0224244d70b90df81643acc WatchSource:0}: Error finding container 4df9ed73329a05017dc55b7b9c9f6cb0af932ea3c0224244d70b90df81643acc: Status 404 returned error can't find the container with id 4df9ed73329a05017dc55b7b9c9f6cb0af932ea3c0224244d70b90df81643acc Nov 04 01:02:08.468422 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.468396 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:08.470269 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.470241 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4qqzm" Nov 04 01:02:08.477083 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.477063 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:08.479268 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.479245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:08.481710 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.481681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:08.481958 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.481935 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b221911_1895_4385_b628_c84f0caefe17.slice/crio-94ac29b2406ec6bde6bb1c7405812179c0834307c47b7eaaafdec5a8721b709d WatchSource:0}: Error finding container 94ac29b2406ec6bde6bb1c7405812179c0834307c47b7eaaafdec5a8721b709d: Status 404 returned error can't find the container with id 94ac29b2406ec6bde6bb1c7405812179c0834307c47b7eaaafdec5a8721b709d Nov 04 01:02:08.482598 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.482573 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63e384d_b749_410c_a347_b35ac0f4a318.slice/crio-b42030284b7dc8c7d5373e1159e34683d4a7341fd650818210b2aa5f2cc079f6 WatchSource:0}: Error finding container b42030284b7dc8c7d5373e1159e34683d4a7341fd650818210b2aa5f2cc079f6: Status 404 returned error can't find the container with id b42030284b7dc8c7d5373e1159e34683d4a7341fd650818210b2aa5f2cc079f6 Nov 04 01:02:08.489060 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.489041 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1492af_3805_4346_8db7_55e5db78f06d.slice/crio-b067a796dcd6b02ac807f5b00a328fd7015f250fcdf55b5838feff0e156fdcbd WatchSource:0}: Error finding container b067a796dcd6b02ac807f5b00a328fd7015f250fcdf55b5838feff0e156fdcbd: Status 404 returned error can't find the container with id b067a796dcd6b02ac807f5b00a328fd7015f250fcdf55b5838feff0e156fdcbd Nov 04 01:02:08.492258 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.492235 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c61d96e_9340_453b_854a_944a058cac37.slice/crio-e869d736d1262dc66d947599f3e14635df7f6f91603359f0ea4666a1a49ccfce WatchSource:0}: Error finding container e869d736d1262dc66d947599f3e14635df7f6f91603359f0ea4666a1a49ccfce: Status 404 returned error can't find the container with id e869d736d1262dc66d947599f3e14635df7f6f91603359f0ea4666a1a49ccfce Nov 04 01:02:08.495046 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.495024 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40442e9_f920_48ca_a3bd_8c19c022d168.slice/crio-866e1080fb647236e053db2b205e741051a8f6246bb1b1eeee9c81af2891c2a5 WatchSource:0}: Error finding container 866e1080fb647236e053db2b205e741051a8f6246bb1b1eeee9c81af2891c2a5: Status 404 returned error can't find the container with id 866e1080fb647236e053db2b205e741051a8f6246bb1b1eeee9c81af2891c2a5 Nov 04 01:02:08.499162 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.499140 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:08.510877 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:08.510849 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9f7134_5311_413e_8a94_4170b61d71dd.slice/crio-87e438242c514db498adf22f2fba015a99ae1b75e903b3e86ad2701c510718ad WatchSource:0}: Error finding container 87e438242c514db498adf22f2fba015a99ae1b75e903b3e86ad2701c510718ad: Status 404 returned error can't find the container with id 87e438242c514db498adf22f2fba015a99ae1b75e903b3e86ad2701c510718ad Nov 04 01:02:08.852980 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.852862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:08.852980 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.852907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:08.853221 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.853064 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:08.853221 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.853101 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:08.853221 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.853188 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs podName:629a09a7-b384-4b7b-82a0-f91930c909aa nodeName:}" failed. No retries permitted until 2026-11-04 01:02:09.853143637 +0000 UTC m=+4.081798018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs") pod "network-metrics-daemon-pndwj" (UID: "629a09a7-b384-4b7b-82a0-f91930c909aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:08.853221 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.853212 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret podName:389e7a15-7659-450c-9483-d59ea607e94d nodeName:}" failed. No retries permitted until 2026-11-04 01:02:09.853201028 +0000 UTC m=+4.081855402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret") pod "global-pull-secret-syncer-h7zqj" (UID: "389e7a15-7659-450c-9483-d59ea607e94d") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:08.953672 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:08.953641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:08.953871 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.953833 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:02:08.953871 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.953854 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:02:08.953871 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.953864 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bt98j for pod openshift-network-diagnostics/network-check-target-tpsdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:08.954047 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:08.953919 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j podName:def4f36e-3bef-48de-acea-cc0e75f0616a nodeName:}" failed. No retries permitted until 2026-11-04 01:02:09.953904349 +0000 UTC m=+4.182558716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt98j" (UniqueName: "kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j") pod "network-check-target-tpsdw" (UID: "def4f36e-3bef-48de-acea-cc0e75f0616a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:09.185731 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.185618 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:57:08 +0000 UTC" deadline="2028-08-07 14:33:45.48940601 +0000 UTC" Nov 04 01:02:09.185731 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.185671 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15421h31m36.303738405s" Nov 04 01:02:09.209780 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.209724 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lkql" event={"ID":"e40442e9-f920-48ca-a3bd-8c19c022d168","Type":"ContainerStarted","Data":"866e1080fb647236e053db2b205e741051a8f6246bb1b1eeee9c81af2891c2a5"} Nov 04 01:02:09.214105 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.214066 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pzkfb" event={"ID":"4c61d96e-9340-453b-854a-944a058cac37","Type":"ContainerStarted","Data":"e869d736d1262dc66d947599f3e14635df7f6f91603359f0ea4666a1a49ccfce"} Nov 04 01:02:09.215747 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.215675 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"b067a796dcd6b02ac807f5b00a328fd7015f250fcdf55b5838feff0e156fdcbd"} Nov 04 01:02:09.217487 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.217442 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4qqzm" event={"ID":"e63e384d-b749-410c-a347-b35ac0f4a318","Type":"ContainerStarted","Data":"b42030284b7dc8c7d5373e1159e34683d4a7341fd650818210b2aa5f2cc079f6"} Nov 04 01:02:09.218565 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.218526 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" event={"ID":"0b221911-1895-4385-b628-c84f0caefe17","Type":"ContainerStarted","Data":"94ac29b2406ec6bde6bb1c7405812179c0834307c47b7eaaafdec5a8721b709d"} Nov 04 01:02:09.219748 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.219725 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" event={"ID":"fc3b56a9-d22c-436d-bb4d-d492b9a385f4","Type":"ContainerStarted","Data":"4df9ed73329a05017dc55b7b9c9f6cb0af932ea3c0224244d70b90df81643acc"} Nov 04 01:02:09.220748 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.220726 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9cpl7" event={"ID":"b58d7d01-e446-4636-8aa6-55302737e478","Type":"ContainerStarted","Data":"40cde1a64fddbed450b03fcede0207e6a378f9a3feb53648633d4bac03e24545"} Nov 04 01:02:09.221996 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.221974 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerStarted","Data":"87e438242c514db498adf22f2fba015a99ae1b75e903b3e86ad2701c510718ad"} Nov 04 01:02:09.223036 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.223015 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cs7zr" event={"ID":"dc46f559-06ba-4445-8048-f29d73b9f168","Type":"ContainerStarted","Data":"6acb85aa87fd466a15e5b0ec37bc65b4ff6c13094b4983dcc3c72da4386ed66b"} Nov 04 01:02:09.863581 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.862070 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:09.863581 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.862128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:09.863581 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.862291 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:09.863581 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.862358 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret podName:389e7a15-7659-450c-9483-d59ea607e94d nodeName:}" failed. No retries permitted until 2026-11-04 01:02:11.862338584 +0000 UTC m=+6.090992967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret") pod "global-pull-secret-syncer-h7zqj" (UID: "389e7a15-7659-450c-9483-d59ea607e94d") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:09.863581 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.863493 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:09.863581 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.863545 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs podName:629a09a7-b384-4b7b-82a0-f91930c909aa nodeName:}" failed. No retries permitted until 2026-11-04 01:02:11.863531822 +0000 UTC m=+6.092186204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs") pod "network-metrics-daemon-pndwj" (UID: "629a09a7-b384-4b7b-82a0-f91930c909aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:09.963402 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:09.962554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:09.963402 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.962847 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:02:09.963402 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.962871 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:02:09.963402 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.962887 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bt98j for pod openshift-network-diagnostics/network-check-target-tpsdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:09.963402 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:09.962951 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j podName:def4f36e-3bef-48de-acea-cc0e75f0616a nodeName:}" failed. No retries permitted until 2026-11-04 01:02:11.962930986 +0000 UTC m=+6.191585371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt98j" (UniqueName: "kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j") pod "network-check-target-tpsdw" (UID: "def4f36e-3bef-48de-acea-cc0e75f0616a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:10.186596 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:10.186499 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:57:08 +0000 UTC" deadline="2028-05-05 16:02:41.346639296 +0000 UTC" Nov 04 01:02:10.186596 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:10.186547 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13167h0m31.160097754s" Nov 04 01:02:10.203836 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:10.203768 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:10.204001 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:10.203909 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:10.204403 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:10.204349 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:10.204502 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:10.204480 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:10.204882 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:10.204557 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:10.204882 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:10.204639 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:11.883472 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:11.883202 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:11.884082 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:11.883503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:11.884082 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.883514 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:11.884082 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.883962 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs podName:629a09a7-b384-4b7b-82a0-f91930c909aa nodeName:}" failed. No retries permitted until 2026-11-04 01:02:15.883571665 +0000 UTC m=+10.112226034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs") pod "network-metrics-daemon-pndwj" (UID: "629a09a7-b384-4b7b-82a0-f91930c909aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:11.885497 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.885470 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:11.885640 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.885563 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret podName:389e7a15-7659-450c-9483-d59ea607e94d nodeName:}" failed. No retries permitted until 2026-11-04 01:02:15.88554514 +0000 UTC m=+10.114199507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret") pod "global-pull-secret-syncer-h7zqj" (UID: "389e7a15-7659-450c-9483-d59ea607e94d") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:11.984664 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:11.984619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:11.984928 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.984909 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:02:11.985006 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.984937 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:02:11.985006 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.984952 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bt98j for pod openshift-network-diagnostics/network-check-target-tpsdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:11.985115 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:11.985022 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j podName:def4f36e-3bef-48de-acea-cc0e75f0616a nodeName:}" failed. No retries permitted until 2026-11-04 01:02:15.985000844 +0000 UTC m=+10.213655212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt98j" (UniqueName: "kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j") pod "network-check-target-tpsdw" (UID: "def4f36e-3bef-48de-acea-cc0e75f0616a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:12.203900 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:12.203867 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:12.204083 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:12.204027 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:12.204471 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:12.204450 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:12.204565 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:12.204551 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:12.204628 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:12.204616 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:12.204709 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:12.204689 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:12.243161 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:12.243115 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"916a1141c4b97a1387808b2945274410","Type":"ContainerStarted","Data":"17492a896d6afbd57a5c3a471a9101ff6bb807530d18500718c8096fb41afe31"} Nov 04 01:02:12.259245 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:12.259205 2583 generic.go:356] "Generic (PLEG): container finished" podID="a95ae3556f824717465d708137d91c49" containerID="658691e9af6614963875c3d841d545bfcd658f8c4ccb69458f31ae9bf221508b" exitCode=0 Nov 04 01:02:12.259428 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:12.259268 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"a95ae3556f824717465d708137d91c49","Type":"ContainerDied","Data":"658691e9af6614963875c3d841d545bfcd658f8c4ccb69458f31ae9bf221508b"} Nov 04 01:02:12.261730 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:12.261674 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" podStartSLOduration=4.2616576330000004 podStartE2EDuration="4.261657633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:12.260961909 +0000 UTC m=+6.489616302" watchObservedRunningTime="2026-11-04 01:02:12.261657633 +0000 UTC m=+6.490312024" Nov 04 01:02:13.267717 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:13.267014 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"a95ae3556f824717465d708137d91c49","Type":"ContainerStarted","Data":"8a41fe120aedd6b735b95f9765cd65097b23e12b302be3888bec7de3fb608664"} Nov 04 01:02:13.272488 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:13.271242 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="3fd6ede7b4c759cf141d3ade3437b563099162035ba0bade5e7bcf5a65d51b36" exitCode=0 Nov 04 01:02:13.272488 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:13.272204 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerDied","Data":"3fd6ede7b4c759cf141d3ade3437b563099162035ba0bade5e7bcf5a65d51b36"} Nov 04 01:02:13.291941 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:13.289561 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" podStartSLOduration=5.289543705 podStartE2EDuration="5.289543705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:13.289531054 +0000 UTC m=+7.518185449" watchObservedRunningTime="2026-11-04 01:02:13.289543705 +0000 UTC m=+7.518198098" Nov 04 01:02:14.203865 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:14.203825 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:14.204062 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:14.203969 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:14.204456 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:14.204429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:14.204588 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:14.204502 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:14.204646 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:14.204608 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:14.204737 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:14.204714 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:14.277324 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:14.277288 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cs7zr" event={"ID":"dc46f559-06ba-4445-8048-f29d73b9f168","Type":"ContainerStarted","Data":"54a8e1b977ad581f54a2b18aa65debf6db249c8deffa4ed6b7c09b1484e4081d"} Nov 04 01:02:14.297618 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:14.297557 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cs7zr" podStartSLOduration=3.47377451 podStartE2EDuration="8.297537932s" totalImagesPullingTime="4.823763422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:14.296652178 +0000 UTC m=+8.525306570" watchObservedRunningTime="2026-11-04 01:02:14.297537932 +0000 UTC m=+8.526192323" Nov 04 01:02:15.291147 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:15.290986 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pzkfb" event={"ID":"4c61d96e-9340-453b-854a-944a058cac37","Type":"ContainerStarted","Data":"b2e765441ae2914bd6efd9239315ce5caad0e0c2294db3704e96c67bbdf175ea"} Nov 04 01:02:15.922657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:15.922617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:15.922850 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:15.922675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:15.922850 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:15.922842 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:15.922960 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:15.922944 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret podName:389e7a15-7659-450c-9483-d59ea607e94d nodeName:}" failed. No retries permitted until 2026-11-04 01:02:23.922923665 +0000 UTC m=+18.151578045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret") pod "global-pull-secret-syncer-h7zqj" (UID: "389e7a15-7659-450c-9483-d59ea607e94d") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:15.923411 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:15.923393 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:15.923517 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:15.923465 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs podName:629a09a7-b384-4b7b-82a0-f91930c909aa nodeName:}" failed. No retries permitted until 2026-11-04 01:02:23.923431563 +0000 UTC m=+18.152085945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs") pod "network-metrics-daemon-pndwj" (UID: "629a09a7-b384-4b7b-82a0-f91930c909aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:16.023638 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:16.023600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:16.023831 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:16.023813 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:02:16.023875 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:16.023839 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:02:16.023875 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:16.023852 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bt98j for pod openshift-network-diagnostics/network-check-target-tpsdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:16.023941 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:16.023914 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j podName:def4f36e-3bef-48de-acea-cc0e75f0616a nodeName:}" failed. No retries permitted until 2026-11-04 01:02:24.023896134 +0000 UTC m=+18.252550518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt98j" (UniqueName: "kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j") pod "network-check-target-tpsdw" (UID: "def4f36e-3bef-48de-acea-cc0e75f0616a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:16.211997 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:16.211924 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:16.212146 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:16.212094 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:16.212596 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:16.212573 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:16.212739 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:16.212716 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:16.212806 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:16.212797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:16.212927 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:16.212908 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:16.295729 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:16.295690 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" event={"ID":"fc3b56a9-d22c-436d-bb4d-d492b9a385f4","Type":"ContainerStarted","Data":"fc441847ed6b4653f413ef8d62a008c24ddcf554b2c02ae4fcae5927f4277860"} Nov 04 01:02:17.245416 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:17.245099 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 01:02:17.299799 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:17.299733 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4qqzm" event={"ID":"e63e384d-b749-410c-a347-b35ac0f4a318","Type":"ContainerStarted","Data":"44a2efee191fb79cdf5b0cfe2bb7f05dc5ea270f0ce4720d3a3a2014913c872d"} Nov 04 01:02:17.303919 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:17.303889 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" event={"ID":"fc3b56a9-d22c-436d-bb4d-d492b9a385f4","Type":"ContainerStarted","Data":"157485aa0b232e8f5aa1534129c455a4535886b9ac939395bda312087cd7b8f9"} Nov 04 01:02:17.322011 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:17.321941 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pzkfb" podStartSLOduration=4.478226268 podStartE2EDuration="10.32192284s" totalImagesPullingTime="5.843696572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:15.31112766 +0000 UTC m=+9.539782051" watchObservedRunningTime="2026-11-04 01:02:17.32192284 +0000 UTC m=+11.550577230" Nov 04 01:02:18.192817 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.192652 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T01:02:17.245120202Z","UUID":"bd65697a-d866-4ccd-8a33-d44ffaf6096e","Handler":null,"Name":"","Endpoint":""} Nov 04 01:02:18.194434 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.194236 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 01:02:18.194434 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.194275 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 01:02:18.201628 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.200964 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:18.201628 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:18.201101 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:18.201628 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.201447 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:18.201628 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:18.201524 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:18.201628 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.201559 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:18.201628 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:18.201600 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:18.308263 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.308215 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9cpl7" event={"ID":"b58d7d01-e446-4636-8aa6-55302737e478","Type":"ContainerStarted","Data":"cdc81f861c47bf551e8b04979bedcaac8f84e33cfa55d6bfd69cec7b1e131fe2"} Nov 04 01:02:18.310954 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.310930 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="f256140797fa6228b05993b832d634ae0710ac3007d6531b42adc78fc9384d02" exitCode=0 Nov 04 01:02:18.311474 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.311449 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerDied","Data":"f256140797fa6228b05993b832d634ae0710ac3007d6531b42adc78fc9384d02"} Nov 04 01:02:18.326721 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.326667 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9cpl7" podStartSLOduration=3.349495902 podStartE2EDuration="12.32665341s" totalImagesPullingTime="8.977157508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:18.326226981 +0000 UTC m=+12.554881367" watchObservedRunningTime="2026-11-04 01:02:18.32665341 +0000 UTC m=+12.555307798" Nov 04 01:02:18.327041 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:18.326995 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4qqzm" podStartSLOduration=2.72058442 podStartE2EDuration="11.326984472s" totalImagesPullingTime="8.606400052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:17.322579393 +0000 UTC m=+11.551233781" watchObservedRunningTime="2026-11-04 01:02:18.326984472 +0000 UTC m=+12.555638860" Nov 04 01:02:19.158670 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:19.158461 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:19.159304 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:19.159282 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:19.314622 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:19.314549 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lkql" event={"ID":"e40442e9-f920-48ca-a3bd-8c19c022d168","Type":"ContainerStarted","Data":"1c0bb94775b2da5c13ba46e73d87e2dcbc3e2e74409d320cf1338fa5e53d7396"} Nov 04 01:02:19.318457 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:19.318425 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" event={"ID":"fc3b56a9-d22c-436d-bb4d-d492b9a385f4","Type":"ContainerStarted","Data":"eceae14b43a4113213a88b37ed2da783358a87b44ab39a873ec3ba51cf6d5b49"} Nov 04 01:02:19.332001 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:19.331956 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2lkql" podStartSLOduration=3.300495498 podStartE2EDuration="12.33194431s" totalImagesPullingTime="9.031448812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:19.331431233 +0000 UTC m=+13.560085621" watchObservedRunningTime="2026-11-04 01:02:19.33194431 +0000 UTC m=+13.560598697" Nov 04 01:02:19.352481 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:19.352433 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" podStartSLOduration=3.10758224 podStartE2EDuration="13.352418483s" totalImagesPullingTime="10.244836243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:19.351855366 +0000 UTC m=+13.580509754" watchObservedRunningTime="2026-11-04 01:02:19.352418483 +0000 UTC m=+13.581072874" Nov 04 01:02:20.201537 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:20.201507 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:20.201997 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:20.201964 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:20.202152 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:20.201513 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:20.202152 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:20.202136 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:20.202264 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:20.202167 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:20.202309 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:20.202284 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:20.323295 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:20.323257 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="02d4913cfa0c7564c27596a0f0e832a2f6e7676a16d00c0f4ff05ef1a9dfd5da" exitCode=0 Nov 04 01:02:20.323806 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:20.323355 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerDied","Data":"02d4913cfa0c7564c27596a0f0e832a2f6e7676a16d00c0f4ff05ef1a9dfd5da"} Nov 04 01:02:20.325169 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:20.325145 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" event={"ID":"0b221911-1895-4385-b628-c84f0caefe17","Type":"ContainerStarted","Data":"6e8cda7cbb5a100b6554461f3b48944b487118e47a3680b1beffa2e5192f7c35"} Nov 04 01:02:20.370349 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:20.370286 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" podStartSLOduration=2.132593689 podStartE2EDuration="13.370273709s" totalImagesPullingTime="11.23768002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:20.369593695 +0000 UTC m=+14.598248082" watchObservedRunningTime="2026-11-04 01:02:20.370273709 +0000 UTC m=+14.598928096" Nov 04 01:02:22.201244 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:22.201215 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:22.201977 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:22.201276 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:22.201977 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:22.201280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:22.201977 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:22.201392 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:22.201977 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:22.201476 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:22.201977 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:22.201542 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:22.332144 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:22.332105 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="0663c5d94a866cb12433c06acce225e2cdfe57baa826bf285d6580226f7f3665" exitCode=0 Nov 04 01:02:22.332301 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:22.332161 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerDied","Data":"0663c5d94a866cb12433c06acce225e2cdfe57baa826bf285d6580226f7f3665"} Nov 04 01:02:23.991404 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:23.991357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:23.991785 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:23.991414 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:23.991785 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:23.991524 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:23.991785 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:23.991578 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret podName:389e7a15-7659-450c-9483-d59ea607e94d nodeName:}" failed. No retries permitted until 2026-11-04 01:02:39.991558597 +0000 UTC m=+34.220212975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret") pod "global-pull-secret-syncer-h7zqj" (UID: "389e7a15-7659-450c-9483-d59ea607e94d") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:02:23.991785 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:23.991633 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:23.991785 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:23.991657 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs podName:629a09a7-b384-4b7b-82a0-f91930c909aa nodeName:}" failed. No retries permitted until 2026-11-04 01:02:39.99164963 +0000 UTC m=+34.220304008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs") pod "network-metrics-daemon-pndwj" (UID: "629a09a7-b384-4b7b-82a0-f91930c909aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:02:24.092208 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:24.091986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:24.092396 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:24.092145 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:02:24.092396 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:24.092306 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:02:24.092396 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:24.092326 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bt98j for pod openshift-network-diagnostics/network-check-target-tpsdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:24.092567 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:24.092405 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j podName:def4f36e-3bef-48de-acea-cc0e75f0616a nodeName:}" failed. No retries permitted until 2026-11-04 01:02:40.092384214 +0000 UTC m=+34.321038602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt98j" (UniqueName: "kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j") pod "network-check-target-tpsdw" (UID: "def4f36e-3bef-48de-acea-cc0e75f0616a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:02:24.200603 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:24.200569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:24.200747 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:24.200662 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:24.200792 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:24.200752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:24.200861 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:24.200851 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:24.201006 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:24.200984 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:24.201103 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:24.201059 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:24.729767 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:24.729664 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:24.729923 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:24.729801 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:02:24.732451 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:24.732415 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:25.344988 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.344760 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"d386d59922675b86149be75bc194b7872a809f6c5ad7af7dff3cf526d71bfc90"} Nov 04 01:02:25.345469 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.345076 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"c0c9aeac09ce692ec13795fa7ba670c3b6c58527aa0db23e864977b248edccbf"} Nov 04 01:02:25.345469 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.345137 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"3976c7788c9d70e02fa3f5423ffb49028e2a46cf0099f3c01707817bb28096b9"} Nov 04 01:02:25.345469 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.345182 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"eedf4a52b5258af2ccc91fdf0eb9a40a0e67e43a190631b3a6aae60e194b72c4"} Nov 04 01:02:25.345469 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.345229 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"6df02336ea3b95f18a09b85b0ff9cb6436e6d0852d606eb2f7f0fc626280efc3"} Nov 04 01:02:25.345469 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.345275 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"4e6a2069eb9e810623288d6ae240cfc87ebfecfa397c322345e828658ed830da"} Nov 04 01:02:25.347511 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.347486 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="dc04e63137b3cc2e0573b4b85fd700c5ed4b0d5e6385771d6464beadb32e6e9f" exitCode=0 Nov 04 01:02:25.347607 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:25.347578 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerDied","Data":"dc04e63137b3cc2e0573b4b85fd700c5ed4b0d5e6385771d6464beadb32e6e9f"} Nov 04 01:02:26.201753 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:26.201724 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:26.201930 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:26.201845 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:26.201930 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:26.201886 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:26.201930 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:26.201924 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:26.202023 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:26.201983 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:26.202053 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:26.202023 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:26.351371 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:26.351328 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="2cc6062d8149f2e84e086a2b22d8b0573da836c99112317ad22e99811c2e09a7" exitCode=0 Nov 04 01:02:26.351839 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:26.351378 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerDied","Data":"2cc6062d8149f2e84e086a2b22d8b0573da836c99112317ad22e99811c2e09a7"} Nov 04 01:02:27.357249 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:27.357218 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" event={"ID":"8a9f7134-5311-413e-8a94-4170b61d71dd","Type":"ContainerStarted","Data":"a6c996ce873923b574182e19292e602846141e4eb91422c4d3611c2ee3ff7adf"} Nov 04 01:02:27.360260 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:27.360238 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"6ba4705c1204c43dec1cd7509791a8096fe7d1bf71ad6f842385be12187c3f54"} Nov 04 01:02:27.383650 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:27.383596 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4kzpr" podStartSLOduration=7.533503219 podStartE2EDuration="20.38358549s" totalImagesPullingTime="12.850082271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:27.383397152 +0000 UTC m=+21.612051539" watchObservedRunningTime="2026-11-04 01:02:27.38358549 +0000 UTC m=+21.612239876" Nov 04 01:02:28.201442 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:28.201412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:28.201442 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:28.201428 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:28.201704 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:28.201525 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:28.201704 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:28.201581 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:28.201704 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:28.201636 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:28.201837 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:28.201720 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:30.200843 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.200813 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:30.200843 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.200841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:30.201615 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.200816 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:30.201615 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:30.200917 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:30.201615 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:30.200999 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:30.201615 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:30.201065 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:30.370126 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.370094 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" event={"ID":"7d1492af-3805-4346-8db7-55e5db78f06d","Type":"ContainerStarted","Data":"ba0c9490dc8e9aa0371ca20cd1155d4241441fdd6a29e1f7d3e23cc7aa7d35e5"} Nov 04 01:02:30.370467 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.370441 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:30.370587 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.370520 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:30.370587 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.370569 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:30.385473 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.385450 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:30.385585 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.385540 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:30.404765 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:30.404728 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" podStartSLOduration=8.004031718 podStartE2EDuration="24.40471732s" totalImagesPullingTime="16.400685602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:30.404315495 +0000 UTC m=+24.632969882" watchObservedRunningTime="2026-11-04 01:02:30.40471732 +0000 UTC m=+24.633371707" Nov 04 01:02:31.207051 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:31.207020 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7zqj"] Nov 04 01:02:31.207484 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:31.207136 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:31.207484 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:31.207236 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:31.210524 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:31.210254 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tpsdw"] Nov 04 01:02:31.210524 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:31.210384 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:31.210524 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:31.210485 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:31.210990 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:31.210965 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pndwj"] Nov 04 01:02:31.211118 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:31.211077 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:31.211193 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:31.211173 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:33.201128 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:33.201087 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:33.201526 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:33.201153 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:33.201526 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:33.201121 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:33.201526 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:33.201221 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:33.201526 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:33.201283 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:33.201526 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:33.201368 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:35.201465 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:35.201396 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:35.201800 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:35.201481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:35.201800 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:35.201490 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:35.201800 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:35.201508 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:35.201800 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:35.201588 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:35.201800 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:35.201666 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:37.200959 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:37.200922 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:37.201497 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:37.201020 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:37.201497 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:37.201035 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:37.201497 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:37.201155 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pndwj" podUID="629a09a7-b384-4b7b-82a0-f91930c909aa" Nov 04 01:02:37.201497 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:37.201250 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7zqj" podUID="389e7a15-7659-450c-9483-d59ea607e94d" Nov 04 01:02:37.201497 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:37.201322 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpsdw" podUID="def4f36e-3bef-48de-acea-cc0e75f0616a" Nov 04 01:02:37.583851 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:37.583828 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeReady" Nov 04 01:02:37.583989 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:37.583940 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 01:02:38.958045 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:38.958016 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9cpl7_b58d7d01-e446-4636-8aa6-55302737e478/dns-node-resolver/0.log" Nov 04 01:02:39.201222 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.201182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:39.201455 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.201224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:39.201455 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.201224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:39.206360 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.206329 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 01:02:39.206749 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.206728 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-k9wxw\"" Nov 04 01:02:39.206828 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.206758 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 01:02:39.206828 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.206759 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 01:02:39.206828 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.206787 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 01:02:39.206828 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.206767 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9c9zb\"" Nov 04 01:02:39.206980 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.206961 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5jp5j\"" Nov 04 01:02:39.928110 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:39.928082 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pzkfb_4c61d96e-9340-453b-854a-944a058cac37/node-ca/0.log" Nov 04 01:02:40.013799 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.013768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:40.014235 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.013807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:40.014235 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:40.013911 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 01:02:40.014235 ip-10-0-136-96 kubenswrapper[2583]: E1104 01:02:40.013971 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs podName:629a09a7-b384-4b7b-82a0-f91930c909aa nodeName:}" failed. No retries permitted until 2026-11-04 01:03:12.013954199 +0000 UTC m=+66.242608578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs") pod "network-metrics-daemon-pndwj" (UID: "629a09a7-b384-4b7b-82a0-f91930c909aa") : secret "metrics-daemon-secret" not found Nov 04 01:02:40.017314 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.017292 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:40.114283 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.114248 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:40.116517 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.116497 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:40.176416 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.176389 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:40.193613 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.193562 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:40.560163 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.560132 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tpsdw"] Nov 04 01:02:40.560876 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:40.560856 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7zqj"] Nov 04 01:02:40.562952 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:40.562928 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389e7a15_7659_450c_9483_d59ea607e94d.slice/crio-b79eea1fc4404ba94bdbb6295a22691c2bd8bd6caa60dddb0a1e9db0a177be44 WatchSource:0}: Error finding container b79eea1fc4404ba94bdbb6295a22691c2bd8bd6caa60dddb0a1e9db0a177be44: Status 404 returned error can't find the container with id b79eea1fc4404ba94bdbb6295a22691c2bd8bd6caa60dddb0a1e9db0a177be44 Nov 04 01:02:40.572419 ip-10-0-136-96 kubenswrapper[2583]: W1104 01:02:40.572397 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef4f36e_3bef_48de_acea_cc0e75f0616a.slice/crio-0dc879be557284e30d6aea961d973b4dd9a650c060b5764539cb13e655acbb48 WatchSource:0}: Error finding container 0dc879be557284e30d6aea961d973b4dd9a650c060b5764539cb13e655acbb48: Status 404 returned error can't find the container with id 0dc879be557284e30d6aea961d973b4dd9a650c060b5764539cb13e655acbb48 Nov 04 01:02:41.395114 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:41.395075 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tpsdw" event={"ID":"def4f36e-3bef-48de-acea-cc0e75f0616a","Type":"ContainerStarted","Data":"0dc879be557284e30d6aea961d973b4dd9a650c060b5764539cb13e655acbb48"} Nov 04 01:02:41.404657 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:41.404616 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7zqj" event={"ID":"389e7a15-7659-450c-9483-d59ea607e94d","Type":"ContainerStarted","Data":"b79eea1fc4404ba94bdbb6295a22691c2bd8bd6caa60dddb0a1e9db0a177be44"} Nov 04 01:02:45.414537 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:45.414501 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tpsdw" event={"ID":"def4f36e-3bef-48de-acea-cc0e75f0616a","Type":"ContainerStarted","Data":"a5670e8c2dedbac3267ca0d37089932da82ececf6852928e95097326e18ea95e"} Nov 04 01:02:45.414925 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:45.414619 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:45.433666 ip-10-0-136-96 kubenswrapper[2583]: I1104 01:02:45.433624 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tpsdw" podStartSLOduration=34.662414969 podStartE2EDuration="38.433613074s" totalImagesPullingTime="3.771198105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:45.433449736 +0000 UTC m=+39.662104125" watchObservedRunningTime="2026-11-04 01:02:45.433613074 +0000 UTC m=+39.662267460" Nov 04 01:02:47.924233 ip-10-0-136-96 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 01:02:47.938729 ip-10-0-136-96 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 01:02:47.938951 ip-10-0-136-96 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 01:02:47.939012 ip-10-0-136-96 systemd[1]: kubelet.service: Consumed 2.128s CPU time. Nov 04 01:02:47.949067 ip-10-0-136-96 systemd[1]: Starting Kubernetes Kubelet... Nov 04 01:02:48.004676 ip-10-0-136-96 kubenswrapper[4312]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:02:48.004676 ip-10-0-136-96 kubenswrapper[4312]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 01:02:48.004676 ip-10-0-136-96 kubenswrapper[4312]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:02:48.004676 ip-10-0-136-96 kubenswrapper[4312]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:02:48.007008 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.006933 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.007008 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007006 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.007008 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007011 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007016 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007021 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007025 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007028 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007032 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007036 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007040 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007044 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007048 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007052 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007056 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007060 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007064 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007068 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007073 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007077 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007082 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007085 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007090 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.007217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007094 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007097 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007101 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007107 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007111 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007121 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007126 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007130 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007134 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007139 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007143 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007148 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007153 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007157 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007162 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007166 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007170 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007174 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007179 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.008130 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007183 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007187 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007192 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007199 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007204 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007211 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007215 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007220 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007224 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007228 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007232 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007237 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007241 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007247 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007251 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007256 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007260 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007263 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007268 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007273 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.008987 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007278 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007284 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007290 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007294 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007299 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007304 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007309 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007313 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007317 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007322 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007326 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007347 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007351 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007355 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007360 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007364 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007368 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007374 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007378 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.009897 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007382 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007387 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007391 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007396 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007400 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007405 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007409 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007414 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007418 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007422 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007427 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007431 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007436 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007441 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007445 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007449 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007454 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007459 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007464 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007468 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.010757 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007472 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007476 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007481 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007485 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007489 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.007493 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008825 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008836 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008842 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008847 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008851 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008855 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008860 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008865 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008869 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008874 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008878 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008882 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008887 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008891 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.011693 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008896 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008900 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008904 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008909 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008913 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008918 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008923 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008927 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008932 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008936 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008940 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008944 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008948 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008953 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008957 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008961 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008966 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008970 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008974 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.012606 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008978 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008982 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008986 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008991 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008995 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.008999 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009003 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009007 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009011 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009015 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009020 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009027 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009034 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009039 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009043 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009048 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009052 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009056 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009060 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009065 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.013425 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009070 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009074 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009078 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009083 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009087 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009091 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009095 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009099 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009103 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009108 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009113 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009117 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009122 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009126 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009130 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009134 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009140 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009144 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009149 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009153 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.013970 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009157 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009161 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009166 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009171 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009175 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009179 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009183 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009188 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009192 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009197 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009201 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009205 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009210 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009215 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009219 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009223 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009228 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009232 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009237 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.014531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009241 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009245 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009250 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009258 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009265 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009271 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009276 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009281 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009285 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009289 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009294 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009298 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009302 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.009306 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010148 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010158 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010164 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010169 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010174 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.015046 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010179 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010183 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010188 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010192 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010197 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010201 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010206 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010211 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010215 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010219 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010224 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010229 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010234 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010238 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010243 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010247 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010251 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010256 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010260 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.015545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010265 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010269 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010273 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010277 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010282 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010286 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010291 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010296 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010300 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010305 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010309 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010314 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010318 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010322 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010326 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010349 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010354 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010359 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010363 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010367 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.016018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010372 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010376 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010381 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010386 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010391 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010396 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010401 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010405 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010409 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010413 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010418 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010422 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010426 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010431 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010435 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010439 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010443 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010447 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010452 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010456 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.016524 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010460 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010465 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010469 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010473 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010478 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010483 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010487 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010491 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010495 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010499 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010503 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010508 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010518 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010523 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010527 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010532 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010536 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010542 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010548 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.017240 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010553 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010558 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010563 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010567 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010572 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010576 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010580 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010585 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010589 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010597 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010603 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010608 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010613 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010618 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010622 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010627 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010633 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010637 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010642 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.018084 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010646 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010651 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010655 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.010660 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010781 4312 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010793 4312 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010803 4312 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010810 4312 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010817 4312 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010823 4312 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010829 4312 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010836 4312 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010841 4312 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010847 4312 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010852 4312 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010857 4312 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010862 4312 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010867 4312 flags.go:64] FLAG: --cgroup-root="" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010872 4312 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010876 4312 flags.go:64] FLAG: --client-ca-file="" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010881 4312 flags.go:64] FLAG: --cloud-provider="external" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010886 4312 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010893 4312 flags.go:64] FLAG: --cluster-domain="" Nov 04 01:02:48.018907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010898 4312 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010903 4312 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010908 4312 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010913 4312 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010919 4312 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010924 4312 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010932 4312 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010938 4312 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010943 4312 flags.go:64] FLAG: --contention-profiling="false" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010948 4312 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010953 4312 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010958 4312 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010963 4312 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010970 4312 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010975 4312 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010980 4312 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010985 4312 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010990 4312 flags.go:64] FLAG: --enable-server="true" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.010995 4312 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011003 4312 flags.go:64] FLAG: --event-burst="100" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011008 4312 flags.go:64] FLAG: --event-qps="50" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011013 4312 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011019 4312 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011023 4312 flags.go:64] FLAG: --eviction-hard="" Nov 04 01:02:48.019931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011030 4312 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011035 4312 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011039 4312 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011045 4312 flags.go:64] FLAG: --eviction-soft="" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011049 4312 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011054 4312 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011059 4312 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011064 4312 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011069 4312 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011073 4312 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011078 4312 flags.go:64] FLAG: --feature-gates="" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011084 4312 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011089 4312 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011094 4312 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011101 4312 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011107 4312 flags.go:64] FLAG: --healthz-port="10248" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011112 4312 flags.go:64] FLAG: --help="false" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011117 4312 flags.go:64] FLAG: --hostname-override="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011122 4312 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011127 4312 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011133 4312 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011138 4312 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011144 4312 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 01:02:48.020995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011149 4312 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011154 4312 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011158 4312 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011163 4312 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011168 4312 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011174 4312 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011179 4312 flags.go:64] FLAG: --kube-reserved="" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011183 4312 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011188 4312 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011193 4312 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011198 4312 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011203 4312 flags.go:64] FLAG: --lock-file="" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011207 4312 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011212 4312 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011216 4312 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011225 4312 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011243 4312 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011248 4312 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011253 4312 flags.go:64] FLAG: --logging-format="text" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011258 4312 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011264 4312 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011268 4312 flags.go:64] FLAG: --manifest-url="" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011273 4312 flags.go:64] FLAG: --manifest-url-header="" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011282 4312 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011287 4312 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 01:02:48.022135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011293 4312 flags.go:64] FLAG: --max-pods="110" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011298 4312 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011303 4312 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011307 4312 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011312 4312 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011317 4312 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011322 4312 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011327 4312 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011353 4312 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011358 4312 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011363 4312 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011368 4312 flags.go:64] FLAG: --pod-cidr="" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011372 4312 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011378 4312 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011383 4312 flags.go:64] FLAG: --pods-per-core="0" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011387 4312 flags.go:64] FLAG: --port="10250" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011392 4312 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011397 4312 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04de64db023bb9d02" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011403 4312 flags.go:64] FLAG: --qos-reserved="" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011408 4312 flags.go:64] FLAG: --read-only-port="10255" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011413 4312 flags.go:64] FLAG: --register-node="true" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011417 4312 flags.go:64] FLAG: --register-with-taints="" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011423 4312 flags.go:64] FLAG: --registry-burst="10" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011428 4312 flags.go:64] FLAG: --registry-qps="5" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011435 4312 flags.go:64] FLAG: --reserved-cpus="" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011440 4312 flags.go:64] FLAG: --reserved-memory="" Nov 04 01:02:48.023227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011447 4312 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011452 4312 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011457 4312 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011461 4312 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011468 4312 flags.go:64] FLAG: --runonce="false" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011474 4312 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011479 4312 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011484 4312 flags.go:64] FLAG: --seccomp-default="false" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011489 4312 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011494 4312 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011499 4312 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011504 4312 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011509 4312 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011514 4312 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011519 4312 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011524 4312 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011529 4312 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011533 4312 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011539 4312 flags.go:64] FLAG: --system-cgroups="" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011543 4312 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011552 4312 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011557 4312 flags.go:64] FLAG: --tls-cert-file="" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011562 4312 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011575 4312 flags.go:64] FLAG: --tls-min-version="" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011580 4312 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 01:02:48.024401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011584 4312 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011589 4312 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011593 4312 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011598 4312 flags.go:64] FLAG: --v="2" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011605 4312 flags.go:64] FLAG: --version="false" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011613 4312 flags.go:64] FLAG: --vmodule="" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011619 4312 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.011625 4312 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011840 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011847 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011851 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011859 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011863 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011868 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011872 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011876 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011881 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011885 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011889 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011894 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011899 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.025014 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011903 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011907 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011912 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011916 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011920 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011924 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011928 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011933 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011938 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011943 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011947 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011951 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011955 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011960 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011964 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011968 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011975 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011980 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011984 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011989 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.025551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011994 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.011998 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012004 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012008 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012012 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012017 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012021 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012026 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012030 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012035 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012039 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012044 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012048 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012053 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012057 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012061 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012066 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012070 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012074 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.026042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012079 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012084 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012089 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012093 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012097 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012101 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012105 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012110 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012114 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012120 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012125 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012129 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012133 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012140 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012147 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012155 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012161 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012166 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012170 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.026601 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012175 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012179 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012184 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012188 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012193 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012197 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012202 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012206 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012210 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012215 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012220 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012224 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012229 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012233 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012238 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012243 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012247 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012252 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012256 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.027064 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012262 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012269 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012274 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012280 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012285 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012290 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012294 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012298 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012303 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012309 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012313 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012317 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012322 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012326 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012346 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012351 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.027537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.012392 4312 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012453 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012460 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012466 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012470 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012475 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012479 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012483 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012487 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012492 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012496 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012500 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012505 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012510 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012514 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012519 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012523 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012527 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012531 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.027935 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012536 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012542 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012546 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012550 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012555 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012560 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012567 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012571 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012575 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012580 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012584 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012589 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012593 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012597 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012601 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012605 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012612 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012618 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012622 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.028539 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012628 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012633 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012637 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012642 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012647 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012652 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012658 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012662 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012667 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012672 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012676 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012681 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012685 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012689 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012695 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012700 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012704 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012708 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012712 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012717 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.029012 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012721 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012726 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012732 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012739 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012743 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012747 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012752 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012756 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012760 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012764 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012769 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012773 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012777 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012782 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012786 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012790 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012795 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012799 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012803 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012807 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.029558 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012812 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012817 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012821 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012825 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012829 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012833 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012839 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012844 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012848 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012852 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012857 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012861 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012866 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012870 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012874 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012878 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012883 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012887 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012891 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012895 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.030042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012901 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012906 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012910 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012915 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012919 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012923 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012927 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012931 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.012936 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.018418 4312 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.018439 4312 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.018472 4312 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.018480 4312 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018572 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018581 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018587 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018592 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018596 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018602 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.030560 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018607 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018612 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018616 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018621 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018625 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018630 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018634 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018641 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018648 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018653 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018659 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018664 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018668 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018673 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018677 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018682 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018686 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018691 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018696 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.031053 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018700 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018705 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018709 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018714 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018718 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018722 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018727 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018731 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018736 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018740 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018745 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018749 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018753 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018757 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018762 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018766 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018770 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018775 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018782 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.031529 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018788 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018793 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018799 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018803 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018808 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018812 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018816 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018821 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018827 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018832 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018836 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018840 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018844 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018849 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018853 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018857 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018862 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018866 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018871 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018876 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.031999 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018880 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018884 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018888 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018893 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018897 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018902 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018906 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018910 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018914 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018918 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018922 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018927 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018931 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018935 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018940 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018947 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018951 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018956 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018960 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018965 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.032488 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018969 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018973 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018979 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018983 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018987 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018992 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.018996 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019001 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019005 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019009 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019014 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019018 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019022 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019026 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019030 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019035 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019039 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019043 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019048 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019052 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.032953 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019056 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019061 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.019103 4312 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019325 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019350 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019354 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019359 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019363 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019370 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019375 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019380 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019384 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019389 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019394 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019398 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019402 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.033454 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019407 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019411 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019415 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019419 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019423 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019428 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019432 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019437 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019441 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019445 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019450 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019454 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019458 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019462 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019467 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019471 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019475 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019480 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019487 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.033838 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019492 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019498 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019503 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019508 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019513 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019517 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019523 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019527 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019532 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019537 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019541 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019546 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019550 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019554 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019559 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019564 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019575 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019580 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019584 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019588 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.034295 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019592 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019597 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019602 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019606 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019610 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019615 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019619 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019623 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019627 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019631 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019636 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019640 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019645 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019649 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019656 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019661 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019665 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019670 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019674 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019683 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.034792 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019688 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019693 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019697 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019701 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019706 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019710 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019715 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019722 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019728 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019734 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019738 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019743 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019748 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019752 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019756 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019761 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019765 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019770 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019775 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.035269 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019779 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019783 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019788 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019792 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019796 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019800 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019805 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019811 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019815 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019820 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019824 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019829 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019834 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019838 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019843 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.019884 4312 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:02:48.035802 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019935 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019942 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019947 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019952 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019956 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019961 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019967 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019972 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019976 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019981 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019986 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019990 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019994 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.019999 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020003 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020007 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020012 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020016 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020021 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020025 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:02:48.036182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020030 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020034 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020038 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020043 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020047 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020052 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020056 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020089 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020099 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020104 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020110 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020114 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020119 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020123 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020127 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020132 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020136 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020141 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020145 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020149 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:02:48.036705 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020154 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020158 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020163 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020167 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020171 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020176 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020180 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020184 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020188 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020193 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020197 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020202 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020206 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020211 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020215 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020220 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020225 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020231 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020238 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:02:48.037187 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020243 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020249 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020254 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020259 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020264 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020269 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020273 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020278 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020283 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020287 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020292 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020296 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020300 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020306 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020311 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020315 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020319 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020324 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020345 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020350 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:02:48.037658 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020354 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020359 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020363 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020367 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020371 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020375 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020380 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020384 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020389 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020393 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020398 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020402 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020406 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020414 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020418 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020423 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020427 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020433 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020438 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020443 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:02:48.038138 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020447 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020452 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020456 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020460 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020465 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020469 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:48.020476 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.020556 4312 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.022102 4312 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.022175 4312 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.022446 4312 server.go:1014] "Starting client certificate rotation" Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.022582 4312 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.022637 4312 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:57:06 +0000 UTC" deadline="2028-04-16 08:28:11.309242537 +0000 UTC" Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.022665 4312 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12703h25m23.286580538s" Nov 04 01:02:48.038644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.023143 4312 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:02:48.043052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.043030 4312 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:02:48.044539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.044527 4312 log.go:25] "Validated CRI v1 runtime API" Nov 04 01:02:48.046079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.046063 4312 log.go:25] "Validated CRI v1 image API" Nov 04 01:02:48.046827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.046806 4312 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 01:02:48.048296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.048267 4312 fs.go:135] Filesystem UUIDs: map[38ad2dac-9884-4e02-b618-beda4164003b:/dev/nvme0n1p3 52caf3fb-82b1-48b5-b10b-a67a63c9e575:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 01:02:48.048482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.048293 4312 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dc879be557284e30d6aea961d973b4dd9a650c060b5764539cb13e655acbb48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dc879be557284e30d6aea961d973b4dd9a650c060b5764539cb13e655acbb48/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40cde1a64fddbed450b03fcede0207e6a378f9a3feb53648633d4bac03e24545/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40cde1a64fddbed450b03fcede0207e6a378f9a3feb53648633d4bac03e24545/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4df9ed73329a05017dc55b7b9c9f6cb0af932ea3c0224244d70b90df81643acc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4df9ed73329a05017dc55b7b9c9f6cb0af932ea3c0224244d70b90df81643acc/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6acb85aa87fd466a15e5b0ec37bc65b4ff6c13094b4983dcc3c72da4386ed66b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6acb85aa87fd466a15e5b0ec37bc65b4ff6c13094b4983dcc3c72da4386ed66b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/866e1080fb647236e053db2b205e741051a8f6246bb1b1eeee9c81af2891c2a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/866e1080fb647236e053db2b205e741051a8f6246bb1b1eeee9c81af2891c2a5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87e438242c514db498adf22f2fba015a99ae1b75e903b3e86ad2701c510718ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87e438242c514db498adf22f2fba015a99ae1b75e903b3e86ad2701c510718ad/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b067a796dcd6b02ac807f5b00a328fd7015f250fcdf55b5838feff0e156fdcbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b067a796dcd6b02ac807f5b00a328fd7015f250fcdf55b5838feff0e156fdcbd/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b42030284b7dc8c7d5373e1159e34683d4a7341fd650818210b2aa5f2cc079f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b42030284b7dc8c7d5373e1159e34683d4a7341fd650818210b2aa5f2cc079f6/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b79eea1fc4404ba94bdbb6295a22691c2bd8bd6caa60dddb0a1e9db0a177be44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b79eea1fc4404ba94bdbb6295a22691c2bd8bd6caa60dddb0a1e9db0a177be44/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e869d736d1262dc66d947599f3e14635df7f6f91603359f0ea4666a1a49ccfce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e869d736d1262dc66d947599f3e14635df7f6f91603359f0ea4666a1a49ccfce/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~projected/kube-api-access-tddvm:{mountpoint:/var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~projected/kube-api-access-tddvm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/389e7a15-7659-450c-9483-d59ea607e94d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/389e7a15-7659-450c-9483-d59ea607e94d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c61d96e-9340-453b-854a-944a058cac37/volumes/kubernetes.io~projected/kube-api-access-ntf64:{mountpoint:/var/lib/kubelet/pods/4c61d96e-9340-453b-854a-944a058cac37/volumes/kubernetes.io~projected/kube-api-access-ntf64 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/629a09a7-b384-4b7b-82a0-f91930c909aa/volumes/kubernetes.io~projected/kube-api-access-x4g8g:{mountpoint:/var/lib/kubelet/pods/629a09a7-b384-4b7b-82a0-f91930c909aa/volumes/kubernetes.io~projected/kube-api-access-x4g8g major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volumes/kubernetes.io~projected/kube-api-access-sqqzb:{mountpoint:/var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volumes/kubernetes.io~projected/kube-api-access-sqqzb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a9f7134-5311-413e-8a94-4170b61d71dd/volumes/kubernetes.io~projected/kube-api-access-qn69b:{mountpoint:/var/lib/kubelet/pods/8a9f7134-5311-413e-8a94-4170b61d71dd/volumes/kubernetes.io~projected/kube-api-access-qn69b major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b58d7d01-e446-4636-8aa6-55302737e478/volumes/kubernetes.io~projected/kube-api-access-j4lrg:{mountpoint:/var/lib/kubelet/pods/b58d7d01-e446-4636-8aa6-55302737e478/volumes/kubernetes.io~projected/kube-api-access-j4lrg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc46f559-06ba-4445-8048-f29d73b9f168/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/dc46f559-06ba-4445-8048-f29d73b9f168/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/def4f36e-3bef-48de-acea-cc0e75f0616a/volumes/kubernetes.io~projected/kube-api-access-bt98j:{mountpoint:/var/lib/kubelet/pods/def4f36e-3bef-48de-acea-cc0e75f0616a/volumes/kubernetes.io~projected/kube-api-access-bt98j major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e40442e9-f920-48ca-a3bd-8c19c022d168/volumes/kubernetes.io~projected/kube-api-access-g9z5j:{mountpoint:/var/lib/kubelet/pods/e40442e9-f920-48ca-a3bd-8c19c022d168/volumes/kubernetes.io~projected/kube-api-access-g9z5j major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e63e384d-b749-410c-a347-b35ac0f4a318/volumes/kubernetes.io~projected/kube-api-access-8ch9l:{mountpoint:/var/lib/kubelet/pods/e63e384d-b749-410c-a347-b35ac0f4a318/volumes/kubernetes.io~projected/kube-api-access-8ch9l major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3b56a9-d22c-436d-bb4d-d492b9a385f4/volumes/kubernetes.io~projected/kube-api-access-p2c7r:{mountpoint:/var/lib/kubelet/pods/fc3b56a9-d22c-436d-bb4d-d492b9a385f4/volumes/kubernetes.io~projected/kube-api-access-p2c7r major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9dbbe0d8ee800b7c42270043894bbe1b139fadd477a2ada8c9117b94e4214667/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d01c0929c82fdedb3fe75594d52d1f7ca1f8c75677b5f46a945cc07c147e5029/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/03d7979e35d750a37bc7cf46fe9ca31a481a7551e0802950749dc0496661eacd/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/1bc3ef63c518265903e6c36d18ecb7cf82939098000afffe53e19c895b6b21c8/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/45c41767cfef619875972d9e710eb5ca7ca9eed331681bfe7e2813d9e834aa14/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cd53666059ea0e980579ce6e5efddf8a4f534b6cdf4aa66499037c7a5b738e1c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9a23ea73891d437e7c2dc17f8d72b36e5132b33ebc7f94428471d5ffb8cc76d4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/38dd3c3752931735b5b6aec3f34c953558e06ce1a2d3e76cf400524cd382d3a8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/7ffce1e0b6052456d995dfbf4a607c7a9724a9ecdd39fa8de2e02b70ce0c9bc2/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1a240976b58cf022a604ad07927c6d4ba3582d12eed335e0ba13e98fdc18cc6b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/919cb2bc29e7692c9fc0ec5738842a7c2d0ae01bbf0e4e4b7514d783a9db479c/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/a9362c2db9abd7575a4b796cb2430d1a57532b75fbe63716200bcb8e307eaa24/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2fabd5e6af0912a52eb346b2f154099475aa0869b090c3520186694739c221ad/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4791b5d21e7536fa5c71af33bf0e1ac365e8f748460bc656aeff922812b94b36/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7a23069261f6cd4c4b07a3a5224323f57d55752ca4045f7b827bbf5d1dd3d4f0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/43c332aec546104ab2bc4849ca5f4d8b1b8592c3fa509d994b16e56229cde03b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7bfd3492c881f335da43571eb04fce02a1d48bb38b9ff8062272c10235f7a747/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/420a9dae832b27dbb845b729086f2520d04c9e535e47e5995f7dc1e39e02df22/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0dfcd96b3c540d73c78bca22f9ecd31e14717b181653278aca19b0e87a904138/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/37bab0fcb062c6d6c5e068a460bc052e626e5ae33458b8e9ca9b411336ffa743/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3abce942656a897cf73451f77a7c9db4dab1209959adea2cb65b46665a677216/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/a539f1eff44f5479085a4d458050cd7563ded157531e76b2dfb82eaa152fc414/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/e4663e6dce301144c3e6f029aa860bad14ceef7fd02ccde42489b42a4d2ebc98/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/974a033273015f7c06a99daacadd37b8f6bae083f6f4bdb6e3127346c2d2fefb/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/828dee6d5ab8f85dab2bb0ba81584cf92f732928cd92a3e05d54b2a5d95a8043/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/f9b0f6365a20a8d9eaec39acebe49b370e74c4a321a97b5a18ddfd5c41f991a2/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d5b4cba3b19cbfebd1c53543df8e79ce0750fa2d5d12716f76ed67f346a66678/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d083336fcd867249ef2247327cda436a1870c41ffa51277cb05b79efedb2af9c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/0b458eba77b6169956733807a0c3946c8ab9c76f8979dcc58ca58a46ed1c2f80/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/420063ae6f9601497481e2dc64689516ccd0bcf84eef0076310afbf02238489b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/c4c063d488ccc7b50fcba9aaca3a9278dc0ee16edc628f7627905999797c7e48/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6c839cdcfe888de6aab69da609ec44c7c4b129dd74ca054e0c77690e1861ec69/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2f864b72aa22717e93a9a1802fa9a0a615e9f2e8d46021820b1490cc737aa9b8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/009b35e51530825148d10ae55647b781a4513c153f10e404d255596665615667/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e6b34c505af8cd6ef055f5391e6523893f4798b48f025c7b2169ad6d2122d435/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 01:02:48.055809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.055200 4312 manager.go:217] Machine: {Timestamp:2026-11-04 01:02:48.054310016 +0000 UTC m=+0.077242015 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095998 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db052ba5ee4392b69710381cf99a6 SystemUUID:ec2db052-ba5e-e439-2b69-710381cf99a6 BootID:1b83d85a-840c-431c-a21a-7a61c477ca4c Filesystems:[{Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/4c61d96e-9340-453b-854a-944a058cac37/volumes/kubernetes.io~projected/kube-api-access-ntf64 DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e869d736d1262dc66d947599f3e14635df7f6f91603359f0ea4666a1a49ccfce/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87e438242c514db498adf22f2fba015a99ae1b75e903b3e86ad2701c510718ad/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dc46f559-06ba-4445-8048-f29d73b9f168/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dc879be557284e30d6aea961d973b4dd9a650c060b5764539cb13e655acbb48/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d1492af-3805-4346-8db7-55e5db78f06d/volumes/kubernetes.io~projected/kube-api-access-sqqzb DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/e63e384d-b749-410c-a347-b35ac0f4a318/volumes/kubernetes.io~projected/kube-api-access-8ch9l DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/e40442e9-f920-48ca-a3bd-8c19c022d168/volumes/kubernetes.io~projected/kube-api-access-g9z5j DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6acb85aa87fd466a15e5b0ec37bc65b4ff6c13094b4983dcc3c72da4386ed66b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/def4f36e-3bef-48de-acea-cc0e75f0616a/volumes/kubernetes.io~projected/kube-api-access-bt98j DeviceMajor:0 DeviceMinor:197 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4df9ed73329a05017dc55b7b9c9f6cb0af932ea3c0224244d70b90df81643acc/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3b56a9-d22c-436d-bb4d-d492b9a385f4/volumes/kubernetes.io~projected/kube-api-access-p2c7r DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~projected/kube-api-access-tddvm DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/8a9f7134-5311-413e-8a94-4170b61d71dd/volumes/kubernetes.io~projected/kube-api-access-qn69b DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b58d7d01-e446-4636-8aa6-55302737e478/volumes/kubernetes.io~projected/kube-api-access-j4lrg DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/629a09a7-b384-4b7b-82a0-f91930c909aa/volumes/kubernetes.io~projected/kube-api-access-x4g8g DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/0b221911-1895-4385-b628-c84f0caefe17/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b42030284b7dc8c7d5373e1159e34683d4a7341fd650818210b2aa5f2cc079f6/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b79eea1fc4404ba94bdbb6295a22691c2bd8bd6caa60dddb0a1e9db0a177be44/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/866e1080fb647236e053db2b205e741051a8f6246bb1b1eeee9c81af2891c2a5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40cde1a64fddbed450b03fcede0207e6a378f9a3feb53648633d4bac03e24545/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b067a796dcd6b02ac807f5b00a328fd7015f250fcdf55b5838feff0e156fdcbd/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/389e7a15-7659-450c-9483-d59ea607e94d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0dc879be557284e MacAddress:a6:c1:a5:b6:ec:07 Speed:10000 Mtu:8901} {Name:b79eea1fc4404ba MacAddress:e6:3b:22:f5:23:6c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fc:92:fa:09 Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:13:75:9e:57:7b Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:fc:92:fa:09 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c2:5e:90:a4:33:9e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:1d:3c:c6:c0:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 01:02:48.055962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.055814 4312 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 01:02:48.055962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.055869 4312 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 01:02:48.056149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056122 4312 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 01:02:48.056269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056150 4312 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-96.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 01:02:48.056315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056271 4312 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 01:02:48.056315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056278 4312 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 01:02:48.056315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056287 4312 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:02:48.056315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056305 4312 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:02:48.056442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056410 4312 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 01:02:48.056569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056559 4312 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 01:02:48.056673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056664 4312 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 01:02:48.056706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056699 4312 kubelet.go:493] "Attempting to sync node with API server" Nov 04 01:02:48.056734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056711 4312 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 01:02:48.056734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056721 4312 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 01:02:48.056734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056729 4312 kubelet.go:405] "Adding apiserver pod source" Nov 04 01:02:48.056813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.056737 4312 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 01:02:48.057236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.057224 4312 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 01:02:48.057236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.057237 4312 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:02:48.057681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.057664 4312 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 01:02:48.059362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059327 4312 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 01:02:48.059362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059361 4312 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:02:48.059497 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059483 4312 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 01:02:48.059711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059697 4312 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 01:02:48.059783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059737 4312 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 01:02:48.059873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059862 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 01:02:48.059934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059882 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 01:02:48.059934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059892 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 01:02:48.059934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059903 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 01:02:48.059934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059912 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059938 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059948 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059958 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059969 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059978 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.059997 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.060012 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.060056 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 01:02:48.060106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.060066 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 01:02:48.063491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.063476 4312 server.go:1263] "Started kubelet" Nov 04 01:02:48.063598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.063539 4312 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 01:02:48.063678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.063626 4312 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 01:02:48.063754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.063706 4312 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 01:02:48.064125 ip-10-0-136-96 systemd[1]: Started Kubernetes Kubelet. Nov 04 01:02:48.064466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.064216 4312 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 01:02:48.065479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.065462 4312 server.go:317] "Adding debug handlers to kubelet server" Nov 04 01:02:48.067608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.067543 4312 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 01:02:48.069645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.067935 4312 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 01:02:48.069645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.068126 4312 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 01:02:48.069645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.068269 4312 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 01:02:48.069645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.068625 4312 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 01:02:48.069645 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:48.068877 4312 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-96.us-east-2.compute.internal\" not found" Nov 04 01:02:48.069645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.069071 4312 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:57:08 +0000 UTC" deadline="2028-04-14 05:10:54.795647089 +0000 UTC" Nov 04 01:02:48.069645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.069132 4312 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12652h8m6.726519214s" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.070794 4312 factory.go:55] Registering systemd factory Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.070820 4312 factory.go:223] Registration of the systemd container factory successfully Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.070947 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9f7134-5311-413e-8a94-4170b61d71dd" volumeName="kubernetes.io/projected/8a9f7134-5311-413e-8a94-4170b61d71dd-kube-api-access-qn69b" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.070986 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e63e384d-b749-410c-a347-b35ac0f4a318" volumeName="kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-cni-binary-copy" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071007 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b221911-1895-4385-b628-c84f0caefe17" volumeName="kubernetes.io/projected/0b221911-1895-4385-b628-c84f0caefe17-kube-api-access-tddvm" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071025 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c61d96e-9340-453b-854a-944a058cac37" volumeName="kubernetes.io/configmap/4c61d96e-9340-453b-854a-944a058cac37-serviceca" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071044 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d1492af-3805-4346-8db7-55e5db78f06d" volumeName="kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-config" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071060 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d1492af-3805-4346-8db7-55e5db78f06d" volumeName="kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-script-lib" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071081 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b58d7d01-e446-4636-8aa6-55302737e478" volumeName="kubernetes.io/empty-dir/b58d7d01-e446-4636-8aa6-55302737e478-tmp-dir" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071094 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b58d7d01-e446-4636-8aa6-55302737e478" volumeName="kubernetes.io/projected/b58d7d01-e446-4636-8aa6-55302737e478-kube-api-access-j4lrg" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071178 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b221911-1895-4385-b628-c84f0caefe17" volumeName="kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-tmp" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071197 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d1492af-3805-4346-8db7-55e5db78f06d" volumeName="kubernetes.io/projected/7d1492af-3805-4346-8db7-55e5db78f06d-kube-api-access-sqqzb" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071210 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d1492af-3805-4346-8db7-55e5db78f06d" volumeName="kubernetes.io/secret/7d1492af-3805-4346-8db7-55e5db78f06d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071229 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9f7134-5311-413e-8a94-4170b61d71dd" volumeName="kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-binary-copy" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071244 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc46f559-06ba-4445-8048-f29d73b9f168" volumeName="kubernetes.io/configmap/dc46f559-06ba-4445-8048-f29d73b9f168-konnectivity-ca" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071261 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="def4f36e-3bef-48de-acea-cc0e75f0616a" volumeName="kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071281 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40442e9-f920-48ca-a3bd-8c19c022d168" volumeName="kubernetes.io/configmap/e40442e9-f920-48ca-a3bd-8c19c022d168-iptables-alerter-script" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071298 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40442e9-f920-48ca-a3bd-8c19c022d168" volumeName="kubernetes.io/projected/e40442e9-f920-48ca-a3bd-8c19c022d168-kube-api-access-g9z5j" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071343 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e7a15-7659-450c-9483-d59ea607e94d" volumeName="kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071185 4312 factory.go:153] Registering CRI-O factory Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071369 4312 factory.go:223] Registration of the crio container factory successfully Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071383 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c61d96e-9340-453b-854a-944a058cac37" volumeName="kubernetes.io/projected/4c61d96e-9340-453b-854a-944a058cac37-kube-api-access-ntf64" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071411 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="629a09a7-b384-4b7b-82a0-f91930c909aa" volumeName="kubernetes.io/projected/629a09a7-b384-4b7b-82a0-f91930c909aa-kube-api-access-x4g8g" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071435 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9f7134-5311-413e-8a94-4170b61d71dd" volumeName="kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071448 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc46f559-06ba-4445-8048-f29d73b9f168" volumeName="kubernetes.io/secret/dc46f559-06ba-4445-8048-f29d73b9f168-agent-certs" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071465 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e63e384d-b749-410c-a347-b35ac0f4a318" volumeName="kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-multus-daemon-config" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071491 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e63e384d-b749-410c-a347-b35ac0f4a318" volumeName="kubernetes.io/projected/e63e384d-b749-410c-a347-b35ac0f4a318-kube-api-access-8ch9l" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071527 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3b56a9-d22c-436d-bb4d-d492b9a385f4" volumeName="kubernetes.io/projected/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kube-api-access-p2c7r" seLinuxMountContext="" Nov 04 01:02:48.071558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071572 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b221911-1895-4385-b628-c84f0caefe17" volumeName="kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-etc-tuned" seLinuxMountContext="" Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071601 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d1492af-3805-4346-8db7-55e5db78f06d" volumeName="kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-env-overrides" seLinuxMountContext="" Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071619 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9f7134-5311-413e-8a94-4170b61d71dd" volumeName="kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071633 4312 reconstruct.go:97] "Volume reconstruction finished" Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071642 4312 reconciler.go:29] "Reconciler: start to sync state" Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071437 4312 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071826 4312 factory.go:103] Registering Raw factory Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.071844 4312 manager.go:1199] Started watching for new ooms in manager Nov 04 01:02:48.072883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.072504 4312 manager.go:319] Starting recovery of all containers Nov 04 01:02:48.078108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.078091 4312 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 01:02:48.078191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.078126 4312 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:48.078542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.078511 4312 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:48.081504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.081477 4312 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:48.098501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.098476 4312 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 01:02:48.099866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.099842 4312 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 01:02:48.099866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.099862 4312 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 01:02:48.100025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.099881 4312 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 01:02:48.100025 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:48.099919 4312 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 01:02:48.104075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.104054 4312 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:02:48.117320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.117291 4312 generic.go:356] "Generic (PLEG): container finished" podID="a95ae3556f824717465d708137d91c49" containerID="658691e9af6614963875c3d841d545bfcd658f8c4ccb69458f31ae9bf221508b" exitCode=0 Nov 04 01:02:48.122178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.122157 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="2cc6062d8149f2e84e086a2b22d8b0573da836c99112317ad22e99811c2e09a7" exitCode=0 Nov 04 01:02:48.122178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.122176 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="dc04e63137b3cc2e0573b4b85fd700c5ed4b0d5e6385771d6464beadb32e6e9f" exitCode=0 Nov 04 01:02:48.122298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.122183 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="0663c5d94a866cb12433c06acce225e2cdfe57baa826bf285d6580226f7f3665" exitCode=0 Nov 04 01:02:48.122298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.122188 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="02d4913cfa0c7564c27596a0f0e832a2f6e7676a16d00c0f4ff05ef1a9dfd5da" exitCode=0 Nov 04 01:02:48.122298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.122193 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="f256140797fa6228b05993b832d634ae0710ac3007d6531b42adc78fc9384d02" exitCode=0 Nov 04 01:02:48.122298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.122198 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a9f7134-5311-413e-8a94-4170b61d71dd" containerID="3fd6ede7b4c759cf141d3ade3437b563099162035ba0bade5e7bcf5a65d51b36" exitCode=0 Nov 04 01:02:48.132375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.132359 4312 manager.go:324] Recovery completed Nov 04 01:02:48.145882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.145858 4312 cpu_manager.go:226] "Starting" policy="none" Nov 04 01:02:48.145882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.145875 4312 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 01:02:48.146022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.145894 4312 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 01:02:48.146072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.146044 4312 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 01:02:48.146135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.146059 4312 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 01:02:48.146135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.146082 4312 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 01:02:48.146135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.146091 4312 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 01:02:48.146135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.146098 4312 policy_none.go:50] "Start" Nov 04 01:02:48.146135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.146110 4312 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 01:02:48.146135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.146122 4312 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 01:02:48.155302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.155286 4312 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 01:02:48.155377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.155305 4312 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 01:02:48.155377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.155314 4312 policy_none.go:44] "Start" Nov 04 01:02:48.159136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.159112 4312 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 01:02:48.162624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.162612 4312 manager.go:341] "Starting Device Plugin manager" Nov 04 01:02:48.163292 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:48.162635 4312 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 01:02:48.163371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.163297 4312 server.go:87] "Starting device plugin registration server" Nov 04 01:02:48.163640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.163628 4312 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 01:02:48.163672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.163643 4312 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 01:02:48.163758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.163739 4312 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 01:02:48.163906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.163871 4312 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 01:02:48.163997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.163915 4312 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 01:02:48.163997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.163924 4312 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 01:02:48.164105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.164000 4312 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T01:02:48.163902897Z","UUID":"e04e63c5-2528-4d14-9ea1-a4f1a4fe250f","Handler":null,"Name":"","Endpoint":""} Nov 04 01:02:48.165123 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:48.165055 4312 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 01:02:48.165654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.165636 4312 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 01:02:48.165757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.165669 4312 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 01:02:48.200861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.200790 4312 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal"] Nov 04 01:02:48.201296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.201252 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"916a1141c4b97a1387808b2945274410","Type":"ContainerStarted","Data":"17492a896d6afbd57a5c3a471a9101ff6bb807530d18500718c8096fb41afe31"} Nov 04 01:02:48.201391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.201314 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"916a1141c4b97a1387808b2945274410","Type":"ContainerStarted","Data":"00d3a2a32309017869ea2ee126e4287adae7f8e353d25e6399dfdae92c440e39"} Nov 04 01:02:48.201391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.201355 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"a95ae3556f824717465d708137d91c49","Type":"ContainerStarted","Data":"8a41fe120aedd6b735b95f9765cd65097b23e12b302be3888bec7de3fb608664"} Nov 04 01:02:48.201391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.201376 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"a95ae3556f824717465d708137d91c49","Type":"ContainerDied","Data":"658691e9af6614963875c3d841d545bfcd658f8c4ccb69458f31ae9bf221508b"} Nov 04 01:02:48.201551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.201395 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" event={"ID":"a95ae3556f824717465d708137d91c49","Type":"ContainerStarted","Data":"557cf0e86ba66399b8b59e73bd64082161e19848a7711238e11285d2bf871e8e"} Nov 04 01:02:48.201551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.201479 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.201551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.201511 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.211475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.211462 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:02:48.211527 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:48.211503 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.211979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.211968 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:02:48.212027 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:48.211992 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.264118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.264096 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:02:48.265571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.265556 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:02:48.265653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.265585 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:02:48.265653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.265600 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:02:48.265653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.265636 4312 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.278245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.278228 4312 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.278290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.278276 4312 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.278382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.278362 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.278414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.278393 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/916a1141c4b97a1387808b2945274410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"916a1141c4b97a1387808b2945274410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.278457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.278410 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.379411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.379379 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/916a1141c4b97a1387808b2945274410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"916a1141c4b97a1387808b2945274410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.379411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.379413 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.379632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.379432 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.379632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.379480 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.379632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.379501 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95ae3556f824717465d708137d91c49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"a95ae3556f824717465d708137d91c49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:48.379632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:48.379501 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/916a1141c4b97a1387808b2945274410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" (UID: \"916a1141c4b97a1387808b2945274410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:49.056930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.056895 4312 apiserver.go:52] "Watching apiserver" Nov 04 01:02:49.067667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.067639 4312 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 01:02:49.067978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.067960 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-fqgxv","kube-system/global-pull-secret-syncer-h7zqj","openshift-cluster-node-tuning-operator/tuned-4zdt9","openshift-dns/node-resolver-9cpl7","openshift-multus/multus-4qqzm","openshift-multus/multus-additional-cni-plugins-4kzpr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal","kube-system/konnectivity-agent-cs7zr","openshift-network-diagnostics/network-check-target-tpsdw","openshift-network-operator/iptables-alerter-2lkql","kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b","openshift-image-registry/node-ca-pzkfb","openshift-multus/network-metrics-daemon-pndwj"] Nov 04 01:02:49.073563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.073533 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.073671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.073536 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 01:02:49.073671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.073635 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.073795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.073711 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 01:02:49.073918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.073893 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 01:02:49.074128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.074108 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 01:02:49.074222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.074209 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:49.074399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.074376 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 01:02:49.074530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.074511 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n5n45\"" Nov 04 01:02:49.075196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.075132 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.075196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.075167 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhcr5\"" Nov 04 01:02:49.075196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.075173 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.075390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.075243 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dh9vh\"" Nov 04 01:02:49.075390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.075284 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 01:02:49.075558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.075545 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.076425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.076412 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pvmvv\"" Nov 04 01:02:49.076701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.076687 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.077024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077010 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq94r\"" Nov 04 01:02:49.077151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077070 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h8pc2\"" Nov 04 01:02:49.077151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077074 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 01:02:49.077151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077122 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.077151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077142 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cfm9p\"" Nov 04 01:02:49.077321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077205 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 01:02:49.077321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077213 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-k9wxw\"" Nov 04 01:02:49.077321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077224 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.077321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077274 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.077467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.077329 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 01:02:49.078565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078553 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.078626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078616 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 01:02:49.078855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078833 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.078925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078859 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 01:02:49.078925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078860 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.078925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078861 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 01:02:49.078925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078916 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.079121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.078956 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.079121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.079014 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78n7j\"" Nov 04 01:02:49.079121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.079093 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 01:02:49.079235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.079163 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9c9zb\"" Nov 04 01:02:49.079235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.079191 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 01:02:49.079856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.079844 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5jp5j\"" Nov 04 01:02:49.080158 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.080145 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 01:02:49.080751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.080728 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 01:02:49.080751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.080732 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zw2l8\"" Nov 04 01:02:49.080880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.080756 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 01:02:49.081839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081824 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-etc-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.081877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081849 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-bin\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.081877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081868 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2c7r\" (UniqueName: \"kubernetes.io/projected/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kube-api-access-p2c7r\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.081953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081893 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc46f559-06ba-4445-8048-f29d73b9f168-agent-certs\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:49.081953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081932 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081954 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081974 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-registration-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.082038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.081998 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-device-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.082038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082026 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-sys-fs\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.082185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082042 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-node-log\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082151 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc46f559-06ba-4445-8048-f29d73b9f168-konnectivity-ca\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:49.082261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082204 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-var-lib-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082346 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082315 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-env-overrides\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082350 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc46f559-06ba-4445-8048-f29d73b9f168-agent-certs\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:49.082391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082375 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1492af-3805-4346-8db7-55e5db78f06d-ovn-node-metrics-cert\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082376 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc46f559-06ba-4445-8048-f29d73b9f168-konnectivity-ca\") pod \"konnectivity-agent-cs7zr\" (UID: \"dc46f559-06ba-4445-8048-f29d73b9f168\") " pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:02:49.082537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082420 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-script-lib\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082472 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqzb\" (UniqueName: \"kubernetes.io/projected/7d1492af-3805-4346-8db7-55e5db78f06d-kube-api-access-sqqzb\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082481 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-env-overrides\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082522 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.082731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082592 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-slash\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082623 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-script-lib\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082630 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-netns\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082666 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-netd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082697 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-ovn\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082710 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1492af-3805-4346-8db7-55e5db78f06d-ovn-node-metrics-cert\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082724 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-log-socket\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082753 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082774 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-config\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082802 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-systemd-units\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082940 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1492af-3805-4346-8db7-55e5db78f06d-ovnkube-config\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082966 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-socket-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.083126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.082996 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-kubelet\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.083126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.083015 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.083126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.083048 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-systemd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.092309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.092285 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqzb\" (UniqueName: \"kubernetes.io/projected/7d1492af-3805-4346-8db7-55e5db78f06d-kube-api-access-sqqzb\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.092443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.092427 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2c7r\" (UniqueName: \"kubernetes.io/projected/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kube-api-access-p2c7r\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.131076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.131056 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:49.131206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.131192 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:49.147473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.147450 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:02:49.147560 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:49.147495 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:49.148759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.148742 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:02:49.148858 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:02:49.148791 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.us-east-2.compute.internal" Nov 04 01:02:49.169415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.169395 4312 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 01:02:49.183232 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183213 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.183295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183238 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-registration-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.183295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183255 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-sys-fs\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.183295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183281 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.183453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183319 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.183453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183362 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch9l\" (UniqueName: \"kubernetes.io/projected/e63e384d-b749-410c-a347-b35ac0f4a318-kube-api-access-8ch9l\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.183453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183368 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-registration-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.183453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183382 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:49.183453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183391 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-sys-fs\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.183453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183398 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf64\" (UniqueName: \"kubernetes.io/projected/4c61d96e-9340-453b-854a-944a058cac37-kube-api-access-ntf64\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:49.183453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183447 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-kubernetes\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.183777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183475 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-etc-tuned\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.183777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183502 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-cnibin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.183777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183525 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-var-lib-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.183777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183571 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-etc-tuned\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.183777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183579 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-systemd\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.183777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.183602 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-var-lib-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184013 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-sys\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.184065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184046 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-host\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.184094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184069 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-conf-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.184130 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184099 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-cnibin\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.184130 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184124 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.184184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184150 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-slash\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184194 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.184255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184242 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-slash\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184233 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-netd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184270 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddvm\" (UniqueName: \"kubernetes.io/projected/0b221911-1895-4385-b628-c84f0caefe17-kube-api-access-tddvm\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.184311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184277 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-netd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184303 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-dbus\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.184377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184351 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-netns\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.184437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184374 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-kubelet\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.184465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184453 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-conf\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.184500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184475 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-ovn\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184505 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-log-socket\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184512 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-ovn\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184537 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-socket-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.184608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184555 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-socket-dir-parent\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.184608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184598 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-log-socket\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184612 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-socket-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.184707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184628 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-multus-daemon-config\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.184707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184660 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lrg\" (UniqueName: \"kubernetes.io/projected/b58d7d01-e446-4636-8aa6-55302737e478-kube-api-access-j4lrg\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:49.184814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184732 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-systemd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184889 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-modprobe-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.184902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184892 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-run-systemd\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.184988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184943 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4g8g\" (UniqueName: \"kubernetes.io/projected/629a09a7-b384-4b7b-82a0-f91930c909aa-kube-api-access-x4g8g\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:49.184988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184960 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-multus-daemon-config\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.185061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.184990 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-bin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.185061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185007 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.185061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185035 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9z5j\" (UniqueName: \"kubernetes.io/projected/e40442e9-f920-48ca-a3bd-8c19c022d168-kube-api-access-g9z5j\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:49.185181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185144 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-tmp\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.185181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185159 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.185181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185175 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-multus\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185203 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185224 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b221911-1895-4385-b628-c84f0caefe17-tmp\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185239 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-device-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185260 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185273 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-system-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185290 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c61d96e-9340-453b-854a-944a058cac37-serviceca\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185296 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-device-dir\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.185316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185304 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-node-log\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185351 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185368 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-node-log\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185403 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysconfig\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185460 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-k8s-cni-cncf-io\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185483 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185493 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c61d96e-9340-453b-854a-944a058cac37-host\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185589 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-netns\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185608 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-var-lib-kubelet\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.185633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185623 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e40442e9-f920-48ca-a3bd-8c19c022d168-iptables-alerter-script\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185638 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-run\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185651 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-lib-modules\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185667 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-kubelet-config\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185681 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185697 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-system-cni-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185716 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185742 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-run-netns\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185763 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185782 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-systemd-units\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185825 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-systemd-units\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185863 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185865 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c61d96e-9340-453b-854a-944a058cac37-serviceca\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185882 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e40442e9-f920-48ca-a3bd-8c19c022d168-iptables-alerter-script\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185894 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-etc-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185927 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.185941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185931 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc3b56a9-d22c-436d-bb4d-d492b9a385f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nb9b\" (UID: \"fc3b56a9-d22c-436d-bb4d-d492b9a385f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nb9b" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185957 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-etc-openvswitch\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.185994 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-kubelet\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186057 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-kubelet\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186068 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186100 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186138 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn69b\" (UniqueName: \"kubernetes.io/projected/8a9f7134-5311-413e-8a94-4170b61d71dd-kube-api-access-qn69b\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186164 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58d7d01-e446-4636-8aa6-55302737e478-hosts-file\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186180 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-bin\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186234 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1492af-3805-4346-8db7-55e5db78f06d-host-cni-bin\") pod \"ovnkube-node-fqgxv\" (UID: \"7d1492af-3805-4346-8db7-55e5db78f06d\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186288 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-etc-kubernetes\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186285 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a9f7134-5311-413e-8a94-4170b61d71dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186307 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186374 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-cni-binary-copy\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186443 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-hostroot\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186487 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-multus-certs\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186515 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58d7d01-e446-4636-8aa6-55302737e478-tmp-dir\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186539 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40442e9-f920-48ca-a3bd-8c19c022d168-host-slash\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186544 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63e384d-b749-410c-a347-b35ac0f4a318-cni-binary-copy\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.186620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186576 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/389e7a15-7659-450c-9483-d59ea607e94d-original-pull-secret\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.187293 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.186985 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58d7d01-e446-4636-8aa6-55302737e478-tmp-dir\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:49.194112 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.194089 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddvm\" (UniqueName: \"kubernetes.io/projected/0b221911-1895-4385-b628-c84f0caefe17-kube-api-access-tddvm\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.194243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.194221 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntf64\" (UniqueName: \"kubernetes.io/projected/4c61d96e-9340-453b-854a-944a058cac37-kube-api-access-ntf64\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:49.194304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.194221 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ch9l\" (UniqueName: \"kubernetes.io/projected/e63e384d-b749-410c-a347-b35ac0f4a318-kube-api-access-8ch9l\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.194354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.194307 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt98j\" (UniqueName: \"kubernetes.io/projected/def4f36e-3bef-48de-acea-cc0e75f0616a-kube-api-access-bt98j\") pod \"network-check-target-tpsdw\" (UID: \"def4f36e-3bef-48de-acea-cc0e75f0616a\") " pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:49.194657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.194636 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lrg\" (UniqueName: \"kubernetes.io/projected/b58d7d01-e446-4636-8aa6-55302737e478-kube-api-access-j4lrg\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:49.194748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.194731 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4g8g\" (UniqueName: \"kubernetes.io/projected/629a09a7-b384-4b7b-82a0-f91930c909aa-kube-api-access-x4g8g\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:49.194868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.194853 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9z5j\" (UniqueName: \"kubernetes.io/projected/e40442e9-f920-48ca-a3bd-8c19c022d168-kube-api-access-g9z5j\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:49.195749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.195731 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn69b\" (UniqueName: \"kubernetes.io/projected/8a9f7134-5311-413e-8a94-4170b61d71dd-kube-api-access-qn69b\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.287267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287233 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-bin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287265 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287283 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-multus\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287298 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-system-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287315 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287350 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysconfig\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287367 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-bin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287378 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-cni-multus\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287408 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysconfig\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287423 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287423 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-system-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287414 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-k8s-cni-cncf-io\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287458 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287458 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-k8s-cni-cncf-io\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287485 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c61d96e-9340-453b-854a-944a058cac37-host\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287507 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c61d96e-9340-453b-854a-944a058cac37-host\") pod \"node-ca-pzkfb\" (UID: \"4c61d96e-9340-453b-854a-944a058cac37\") " pod="openshift-image-registry/node-ca-pzkfb" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287516 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-var-lib-kubelet\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287533 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-run\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287555 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-lib-modules\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287575 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-kubelet-config\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287589 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287593 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-var-lib-kubelet\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287639 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-system-cni-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.287639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287651 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-run\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287651 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-lib-modules\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287652 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-kubelet-config\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287677 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58d7d01-e446-4636-8aa6-55302737e478-hosts-file\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287702 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-etc-kubernetes\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287714 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-system-cni-dir\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287718 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58d7d01-e446-4636-8aa6-55302737e478-hosts-file\") pod \"node-resolver-9cpl7\" (UID: \"b58d7d01-e446-4636-8aa6-55302737e478\") " pod="openshift-dns/node-resolver-9cpl7" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287726 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-hostroot\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287742 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-etc-kubernetes\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287747 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-multus-certs\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287764 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40442e9-f920-48ca-a3bd-8c19c022d168-host-slash\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287781 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287802 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-kubernetes\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287826 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40442e9-f920-48ca-a3bd-8c19c022d168-host-slash\") pod \"iptables-alerter-2lkql\" (UID: \"e40442e9-f920-48ca-a3bd-8c19c022d168\") " pod="openshift-network-operator/iptables-alerter-2lkql" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287827 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-cnibin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287842 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-hostroot\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287825 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-multus-certs\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287851 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-cni-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287881 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-systemd\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287892 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-kubernetes\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287904 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-sys\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287927 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-host\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287928 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-cnibin\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287940 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-systemd\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287949 4312 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287974 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-sys\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287976 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-host\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.287975 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-conf-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288005 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-conf-dir\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288023 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-cnibin\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288051 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-dbus\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288069 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-netns\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288083 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-kubelet\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288098 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-conf\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288081 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a9f7134-5311-413e-8a94-4170b61d71dd-cnibin\") pod \"multus-additional-cni-plugins-4kzpr\" (UID: \"8a9f7134-5311-413e-8a94-4170b61d71dd\") " pod="openshift-multus/multus-additional-cni-plugins-4kzpr" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288121 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-socket-dir-parent\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288135 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-run-netns\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288143 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-modprobe-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-host-var-lib-kubelet\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288223 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/389e7a15-7659-450c-9483-d59ea607e94d-dbus\") pod \"global-pull-secret-syncer-h7zqj\" (UID: \"389e7a15-7659-450c-9483-d59ea607e94d\") " pod="kube-system/global-pull-secret-syncer-h7zqj" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288230 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-sysctl-conf\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288238 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63e384d-b749-410c-a347-b35ac0f4a318-multus-socket-dir-parent\") pod \"multus-4qqzm\" (UID: \"e63e384d-b749-410c-a347-b35ac0f4a318\") " pod="openshift-multus/multus-4qqzm" Nov 04 01:02:49.288413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.288223 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b221911-1895-4385-b628-c84f0caefe17-etc-modprobe-d\") pod \"tuned-4zdt9\" (UID: \"0b221911-1895-4385-b628-c84f0caefe17\") " pod="openshift-cluster-node-tuning-operator/tuned-4zdt9" Nov 04 01:02:49.289930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.289915 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629a09a7-b384-4b7b-82a0-f91930c909aa-metrics-certs\") pod \"network-metrics-daemon-pndwj\" (UID: \"629a09a7-b384-4b7b-82a0-f91930c909aa\") " pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:49.375412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.375325 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pndwj" Nov 04 01:02:49.671609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.671519 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pndwj"] Nov 04 01:02:49.675013 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:49.674973 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629a09a7_b384_4b7b_82a0_f91930c909aa.slice/crio-05fdf4511b431aa5e899ca0b73b007fb3850ea22e47e15709fb4099667b739f1 WatchSource:0}: Error finding container 05fdf4511b431aa5e899ca0b73b007fb3850ea22e47e15709fb4099667b739f1: Status 404 returned error can't find the container with id 05fdf4511b431aa5e899ca0b73b007fb3850ea22e47e15709fb4099667b739f1 Nov 04 01:02:49.677533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:49.677515 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:02:50.135680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:50.135642 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pndwj" event={"ID":"629a09a7-b384-4b7b-82a0-f91930c909aa","Type":"ContainerStarted","Data":"05fdf4511b431aa5e899ca0b73b007fb3850ea22e47e15709fb4099667b739f1"} Nov 04 01:02:52.142012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:52.141979 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pndwj" event={"ID":"629a09a7-b384-4b7b-82a0-f91930c909aa","Type":"ContainerStarted","Data":"b729fb97706b0a99cc8252616f76a39ddaaf79c24d7cb396b69181027a8d468e"} Nov 04 01:02:52.142385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:52.142028 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pndwj" event={"ID":"629a09a7-b384-4b7b-82a0-f91930c909aa","Type":"ContainerStarted","Data":"36d024d5b0b7b1e0474d4099b6bf7b05e16fbb36be829932eb9f6072891d085a"} Nov 04 01:02:55.540273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:55.540241 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:57.289630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.289595 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52"] Nov 04 01:02:57.291117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.291097 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gvtrm"] Nov 04 01:02:57.291233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.291215 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" Nov 04 01:02:57.292874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.292853 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.301645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.301622 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 04 01:02:57.302078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.302058 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4jm2s\"" Nov 04 01:02:57.303651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.303602 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zr9fn\"" Nov 04 01:02:57.307038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.305699 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52"] Nov 04 01:02:57.307038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.306044 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d6x9n"] Nov 04 01:02:57.307038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.306113 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 01:02:57.307038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.306200 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 01:02:57.308298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.308281 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gvtrm"] Nov 04 01:02:57.308401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.308377 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.314773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.314752 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8wrmb"] Nov 04 01:02:57.316257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.316235 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.317014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.316995 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 01:02:57.321025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.321005 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d6x9n"] Nov 04 01:02:57.323391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.323225 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 01:02:57.323391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.323272 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 01:02:57.330204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.330184 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 01:02:57.330304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.330235 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 01:02:57.331808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.331785 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8wrmb"] Nov 04 01:02:57.338357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.338329 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c2c762ba-d7cd-42e2-8410-1bf6fe2408a3-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hkx52\" (UID: \"c2c762ba-d7cd-42e2-8410-1bf6fe2408a3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" Nov 04 01:02:57.338444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.338422 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89b2\" (UniqueName: \"kubernetes.io/projected/0e4ff47b-fe50-4203-bf00-1d332e02ff6d-kube-api-access-p89b2\") pod \"downloads-6dd789c8b7-gvtrm\" (UID: \"0e4ff47b-fe50-4203-bf00-1d332e02ff6d\") " pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.338489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.338455 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e4ff47b-fe50-4203-bf00-1d332e02ff6d-tmp\") pod \"downloads-6dd789c8b7-gvtrm\" (UID: \"0e4ff47b-fe50-4203-bf00-1d332e02ff6d\") " pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.382548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.382520 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f9sn6"] Nov 04 01:02:57.384317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.384302 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.390951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.390937 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 01:02:57.392301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.392279 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74fd556577-xc96m"] Nov 04 01:02:57.393859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.393829 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.399092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.399074 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f9sn6"] Nov 04 01:02:57.406140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.406117 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 01:02:57.407709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407686 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7ffl4\"" Nov 04 01:02:57.407709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407691 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 01:02:57.407822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407713 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vklqk\"" Nov 04 01:02:57.407822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407761 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 01:02:57.407822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407804 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 01:02:57.407908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407848 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6kj2d\"" Nov 04 01:02:57.407908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407888 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 01:02:57.407969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.407952 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 01:02:57.408739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.408728 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5mjn\"" Nov 04 01:02:57.408780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.408760 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 01:02:57.414581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.414558 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74fd556577-xc96m"] Nov 04 01:02:57.438876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.438855 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbcde58b-e496-453a-9ec1-18007d6bf8b8-cert\") pod \"ingress-canary-8wrmb\" (UID: \"bbcde58b-e496-453a-9ec1-18007d6bf8b8\") " pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.438983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.438882 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-bound-sa-token\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.438983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.438898 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-registry-tls\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.438983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.438915 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b97127b8-d6e3-47f1-bcab-63a53d06c574-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.438983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.438931 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc947\" (UniqueName: \"kubernetes.io/projected/3f5cd56a-808e-4b32-a685-127a21c6cf3d-kube-api-access-jc947\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.439124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439001 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqzm\" (UniqueName: \"kubernetes.io/projected/bbcde58b-e496-453a-9ec1-18007d6bf8b8-kube-api-access-6qqzm\") pod \"ingress-canary-8wrmb\" (UID: \"bbcde58b-e496-453a-9ec1-18007d6bf8b8\") " pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.439124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439027 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f5cd56a-808e-4b32-a685-127a21c6cf3d-metrics-tls\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.439124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439089 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89b2\" (UniqueName: \"kubernetes.io/projected/0e4ff47b-fe50-4203-bf00-1d332e02ff6d-kube-api-access-p89b2\") pod \"downloads-6dd789c8b7-gvtrm\" (UID: \"0e4ff47b-fe50-4203-bf00-1d332e02ff6d\") " pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.439124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439114 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7mn\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-kube-api-access-st7mn\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.439253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439168 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b97127b8-d6e3-47f1-bcab-63a53d06c574-crio-socket\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.439253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439199 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgtd\" (UniqueName: \"kubernetes.io/projected/b97127b8-d6e3-47f1-bcab-63a53d06c574-kube-api-access-xhgtd\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.439323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439261 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e4ff47b-fe50-4203-bf00-1d332e02ff6d-tmp\") pod \"downloads-6dd789c8b7-gvtrm\" (UID: \"0e4ff47b-fe50-4203-bf00-1d332e02ff6d\") " pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.439323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439317 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c2c762ba-d7cd-42e2-8410-1bf6fe2408a3-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hkx52\" (UID: \"c2c762ba-d7cd-42e2-8410-1bf6fe2408a3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" Nov 04 01:02:57.439411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439358 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/705fa7e2-620e-4425-8d01-8f1335e2b13c-installation-pull-secrets\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.439411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439384 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b97127b8-d6e3-47f1-bcab-63a53d06c574-data-volume\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.439484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439421 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b97127b8-d6e3-47f1-bcab-63a53d06c574-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.439484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439458 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5cd56a-808e-4b32-a685-127a21c6cf3d-config-volume\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.439484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439481 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/705fa7e2-620e-4425-8d01-8f1335e2b13c-image-registry-private-configuration\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.439592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439498 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/705fa7e2-620e-4425-8d01-8f1335e2b13c-ca-trust-extracted\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.439592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439521 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/705fa7e2-620e-4425-8d01-8f1335e2b13c-registry-certificates\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.439592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439548 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/705fa7e2-620e-4425-8d01-8f1335e2b13c-trusted-ca\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.439592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439572 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5cd56a-808e-4b32-a685-127a21c6cf3d-tmp-dir\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.439758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.439731 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e4ff47b-fe50-4203-bf00-1d332e02ff6d-tmp\") pod \"downloads-6dd789c8b7-gvtrm\" (UID: \"0e4ff47b-fe50-4203-bf00-1d332e02ff6d\") " pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.441891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.441866 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c2c762ba-d7cd-42e2-8410-1bf6fe2408a3-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hkx52\" (UID: \"c2c762ba-d7cd-42e2-8410-1bf6fe2408a3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" Nov 04 01:02:57.451879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.451854 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89b2\" (UniqueName: \"kubernetes.io/projected/0e4ff47b-fe50-4203-bf00-1d332e02ff6d-kube-api-access-p89b2\") pod \"downloads-6dd789c8b7-gvtrm\" (UID: \"0e4ff47b-fe50-4203-bf00-1d332e02ff6d\") " pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.540079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.539997 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/705fa7e2-620e-4425-8d01-8f1335e2b13c-installation-pull-secrets\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540031 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b97127b8-d6e3-47f1-bcab-63a53d06c574-data-volume\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.540079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540048 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b97127b8-d6e3-47f1-bcab-63a53d06c574-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540092 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5cd56a-808e-4b32-a685-127a21c6cf3d-config-volume\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540123 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/705fa7e2-620e-4425-8d01-8f1335e2b13c-image-registry-private-configuration\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540142 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/705fa7e2-620e-4425-8d01-8f1335e2b13c-ca-trust-extracted\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540158 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/705fa7e2-620e-4425-8d01-8f1335e2b13c-registry-certificates\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540173 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/705fa7e2-620e-4425-8d01-8f1335e2b13c-trusted-ca\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540219 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5cd56a-808e-4b32-a685-127a21c6cf3d-tmp-dir\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540259 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbcde58b-e496-453a-9ec1-18007d6bf8b8-cert\") pod \"ingress-canary-8wrmb\" (UID: \"bbcde58b-e496-453a-9ec1-18007d6bf8b8\") " pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540286 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-bound-sa-token\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540318 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-registry-tls\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540366 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b97127b8-d6e3-47f1-bcab-63a53d06c574-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540389 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc947\" (UniqueName: \"kubernetes.io/projected/3f5cd56a-808e-4b32-a685-127a21c6cf3d-kube-api-access-jc947\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540420 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqzm\" (UniqueName: \"kubernetes.io/projected/bbcde58b-e496-453a-9ec1-18007d6bf8b8-kube-api-access-6qqzm\") pod \"ingress-canary-8wrmb\" (UID: \"bbcde58b-e496-453a-9ec1-18007d6bf8b8\") " pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540444 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f5cd56a-808e-4b32-a685-127a21c6cf3d-metrics-tls\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540496 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7mn\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-kube-api-access-st7mn\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540529 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b97127b8-d6e3-47f1-bcab-63a53d06c574-crio-socket\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540553 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgtd\" (UniqueName: \"kubernetes.io/projected/b97127b8-d6e3-47f1-bcab-63a53d06c574-kube-api-access-xhgtd\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540603 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/705fa7e2-620e-4425-8d01-8f1335e2b13c-ca-trust-extracted\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.540835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540733 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5cd56a-808e-4b32-a685-127a21c6cf3d-config-volume\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.541215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.540964 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b97127b8-d6e3-47f1-bcab-63a53d06c574-crio-socket\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.541215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.541034 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b97127b8-d6e3-47f1-bcab-63a53d06c574-data-volume\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.541313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.541217 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b97127b8-d6e3-47f1-bcab-63a53d06c574-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.541393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.541325 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5cd56a-808e-4b32-a685-127a21c6cf3d-tmp-dir\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.541452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.541422 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/705fa7e2-620e-4425-8d01-8f1335e2b13c-trusted-ca\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.541745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.541715 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/705fa7e2-620e-4425-8d01-8f1335e2b13c-registry-certificates\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.543055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.543024 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b97127b8-d6e3-47f1-bcab-63a53d06c574-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.543519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.543489 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f5cd56a-808e-4b32-a685-127a21c6cf3d-metrics-tls\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.543793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.543770 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/705fa7e2-620e-4425-8d01-8f1335e2b13c-installation-pull-secrets\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.543856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.543774 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/705fa7e2-620e-4425-8d01-8f1335e2b13c-image-registry-private-configuration\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.543856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.543811 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbcde58b-e496-453a-9ec1-18007d6bf8b8-cert\") pod \"ingress-canary-8wrmb\" (UID: \"bbcde58b-e496-453a-9ec1-18007d6bf8b8\") " pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.543970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.543954 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-registry-tls\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.552811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.552785 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-bound-sa-token\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.553223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.553195 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqzm\" (UniqueName: \"kubernetes.io/projected/bbcde58b-e496-453a-9ec1-18007d6bf8b8-kube-api-access-6qqzm\") pod \"ingress-canary-8wrmb\" (UID: \"bbcde58b-e496-453a-9ec1-18007d6bf8b8\") " pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.553402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.553384 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc947\" (UniqueName: \"kubernetes.io/projected/3f5cd56a-808e-4b32-a685-127a21c6cf3d-kube-api-access-jc947\") pod \"dns-default-f9sn6\" (UID: \"3f5cd56a-808e-4b32-a685-127a21c6cf3d\") " pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.553811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.553796 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7mn\" (UniqueName: \"kubernetes.io/projected/705fa7e2-620e-4425-8d01-8f1335e2b13c-kube-api-access-st7mn\") pod \"image-registry-74fd556577-xc96m\" (UID: \"705fa7e2-620e-4425-8d01-8f1335e2b13c\") " pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.553867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.553847 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgtd\" (UniqueName: \"kubernetes.io/projected/b97127b8-d6e3-47f1-bcab-63a53d06c574-kube-api-access-xhgtd\") pod \"insights-runtime-extractor-d6x9n\" (UID: \"b97127b8-d6e3-47f1-bcab-63a53d06c574\") " pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.602664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.602639 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" Nov 04 01:02:57.613240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.613219 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:02:57.623229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.623206 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d6x9n" Nov 04 01:02:57.627028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.627005 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8wrmb" Nov 04 01:02:57.696083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.696055 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f9sn6" Nov 04 01:02:57.704631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.703892 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:57.736713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.736165 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:02:57.882819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.882744 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52"] Nov 04 01:02:57.885776 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:57.885746 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c762ba_d7cd_42e2_8410_1bf6fe2408a3.slice/crio-9733e62bd37f70a316b576f7169504b2abf3764a162562562b364ef842e0b203 WatchSource:0}: Error finding container 9733e62bd37f70a316b576f7169504b2abf3764a162562562b364ef842e0b203: Status 404 returned error can't find the container with id 9733e62bd37f70a316b576f7169504b2abf3764a162562562b364ef842e0b203 Nov 04 01:02:57.901389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.901366 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gvtrm"] Nov 04 01:02:57.904220 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:57.904198 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4ff47b_fe50_4203_bf00_1d332e02ff6d.slice/crio-8e043a0a71fdae6edd15ed537e81079e3f6d3d58fdc2e334e8d535a38b34436e WatchSource:0}: Error finding container 8e043a0a71fdae6edd15ed537e81079e3f6d3d58fdc2e334e8d535a38b34436e: Status 404 returned error can't find the container with id 8e043a0a71fdae6edd15ed537e81079e3f6d3d58fdc2e334e8d535a38b34436e Nov 04 01:02:57.909874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.909851 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d6x9n"] Nov 04 01:02:57.912943 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:57.912920 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97127b8_d6e3_47f1_bcab_63a53d06c574.slice/crio-11f1e643ad3fb11901211236fa90626b3dbbbc9c8a55c7cad87166eb0c63f5aa WatchSource:0}: Error finding container 11f1e643ad3fb11901211236fa90626b3dbbbc9c8a55c7cad87166eb0c63f5aa: Status 404 returned error can't find the container with id 11f1e643ad3fb11901211236fa90626b3dbbbc9c8a55c7cad87166eb0c63f5aa Nov 04 01:02:57.925531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.925508 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8wrmb"] Nov 04 01:02:57.934483 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:57.934457 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcde58b_e496_453a_9ec1_18007d6bf8b8.slice/crio-386bc6722b8b30adbd27f17a9ed87e2763972850c122fa412eeaa4807b10a74a WatchSource:0}: Error finding container 386bc6722b8b30adbd27f17a9ed87e2763972850c122fa412eeaa4807b10a74a: Status 404 returned error can't find the container with id 386bc6722b8b30adbd27f17a9ed87e2763972850c122fa412eeaa4807b10a74a Nov 04 01:02:57.976248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.976224 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f9sn6"] Nov 04 01:02:57.979354 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:57.979314 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5cd56a_808e_4b32_a685_127a21c6cf3d.slice/crio-46f20db4ffa5e7340e437cb0d849273e7882b57ff44dba6480bd943e1d6c4fcc WatchSource:0}: Error finding container 46f20db4ffa5e7340e437cb0d849273e7882b57ff44dba6480bd943e1d6c4fcc: Status 404 returned error can't find the container with id 46f20db4ffa5e7340e437cb0d849273e7882b57ff44dba6480bd943e1d6c4fcc Nov 04 01:02:57.999142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:57.999118 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74fd556577-xc96m"] Nov 04 01:02:58.001909 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:02:58.001886 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705fa7e2_620e_4425_8d01_8f1335e2b13c.slice/crio-55e790826900f26945e6a9199162cb3c8dd69b68643f1cd2d349186fd2d07c7b WatchSource:0}: Error finding container 55e790826900f26945e6a9199162cb3c8dd69b68643f1cd2d349186fd2d07c7b: Status 404 returned error can't find the container with id 55e790826900f26945e6a9199162cb3c8dd69b68643f1cd2d349186fd2d07c7b Nov 04 01:02:58.156094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.156004 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74fd556577-xc96m" event={"ID":"705fa7e2-620e-4425-8d01-8f1335e2b13c","Type":"ContainerStarted","Data":"1479f99299bbd1aa2a2d0e7d3a39820cf9e0a9969bbc38a9c3d01cf20f3baa9d"} Nov 04 01:02:58.156094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.156060 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74fd556577-xc96m" event={"ID":"705fa7e2-620e-4425-8d01-8f1335e2b13c","Type":"ContainerStarted","Data":"55e790826900f26945e6a9199162cb3c8dd69b68643f1cd2d349186fd2d07c7b"} Nov 04 01:02:58.156094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.156084 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:02:58.157626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.157601 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d6x9n" event={"ID":"b97127b8-d6e3-47f1-bcab-63a53d06c574","Type":"ContainerStarted","Data":"f0f6f2a85ac6ee9c0491610a306986ddf38f47d04e899fb96cc12e9defe741ed"} Nov 04 01:02:58.157739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.157652 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d6x9n" event={"ID":"b97127b8-d6e3-47f1-bcab-63a53d06c574","Type":"ContainerStarted","Data":"11f1e643ad3fb11901211236fa90626b3dbbbc9c8a55c7cad87166eb0c63f5aa"} Nov 04 01:02:58.159118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.159086 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8wrmb" event={"ID":"bbcde58b-e496-453a-9ec1-18007d6bf8b8","Type":"ContainerStarted","Data":"386bc6722b8b30adbd27f17a9ed87e2763972850c122fa412eeaa4807b10a74a"} Nov 04 01:02:58.160313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.160292 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f9sn6" event={"ID":"3f5cd56a-808e-4b32-a685-127a21c6cf3d","Type":"ContainerStarted","Data":"46f20db4ffa5e7340e437cb0d849273e7882b57ff44dba6480bd943e1d6c4fcc"} Nov 04 01:02:58.164898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.164870 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gvtrm" event={"ID":"0e4ff47b-fe50-4203-bf00-1d332e02ff6d","Type":"ContainerStarted","Data":"8e043a0a71fdae6edd15ed537e81079e3f6d3d58fdc2e334e8d535a38b34436e"} Nov 04 01:02:58.166156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.166128 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" event={"ID":"c2c762ba-d7cd-42e2-8410-1bf6fe2408a3","Type":"ContainerStarted","Data":"9733e62bd37f70a316b576f7169504b2abf3764a162562562b364ef842e0b203"} Nov 04 01:02:58.182868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:58.182825 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74fd556577-xc96m" podStartSLOduration=1.182813917 podStartE2EDuration="1.182813917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:58.182610147 +0000 UTC m=+10.205542170" watchObservedRunningTime="2026-11-04 01:02:58.182813917 +0000 UTC m=+10.205745964" Nov 04 01:02:59.098666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:59.098590 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:02:59.099075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:59.098760 4312 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:02:59.175006 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:02:59.174937 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d6x9n" event={"ID":"b97127b8-d6e3-47f1-bcab-63a53d06c574","Type":"ContainerStarted","Data":"ef08a9b1c62453030879fbfbb25bcb7605a29ec132d4b300feae13275d78e174"} Nov 04 01:03:00.182920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:00.182877 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" event={"ID":"c2c762ba-d7cd-42e2-8410-1bf6fe2408a3","Type":"ContainerStarted","Data":"693101c60844cead3a83988438bf575978cd0ab69cace6ed8ed974e4e5a87f4d"} Nov 04 01:03:00.202752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:00.202697 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" podStartSLOduration=1.802622889 podStartE2EDuration="3.2026831s" totalImagesPullingTime="1.400060211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:00.202680368 +0000 UTC m=+12.225612400" watchObservedRunningTime="2026-11-04 01:03:00.2026831 +0000 UTC m=+12.225615108" Nov 04 01:03:01.192852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:01.192829 4312 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:03:01.192852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:01.192830 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f9sn6" event={"ID":"3f5cd56a-808e-4b32-a685-127a21c6cf3d","Type":"ContainerStarted","Data":"fff5cfa72afc6494eb1a744bbe075ef3e4545abc9cceeb31ddd009d20853a3f3"} Nov 04 01:03:01.193214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:01.192905 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f9sn6" event={"ID":"3f5cd56a-808e-4b32-a685-127a21c6cf3d","Type":"ContainerStarted","Data":"9a227e40848eb412609704678a2dd0d96dbc4cc5058d038c00765b7a87542263"} Nov 04 01:03:01.193214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:01.192960 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f9sn6" Nov 04 01:03:01.213397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:01.213346 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-f9sn6" podStartSLOduration=1.723676888 podStartE2EDuration="4.213320004s" totalImagesPullingTime="2.489643116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:01.212784664 +0000 UTC m=+13.235716690" watchObservedRunningTime="2026-11-04 01:03:01.213320004 +0000 UTC m=+13.236252008" Nov 04 01:03:02.199887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:02.199849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d6x9n" event={"ID":"b97127b8-d6e3-47f1-bcab-63a53d06c574","Type":"ContainerStarted","Data":"87c7f601b152b0f2c0ad179de2690e13e4d5f36b84c8eedd113e971b8db84e5b"} Nov 04 01:03:02.201801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:02.201769 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8wrmb" event={"ID":"bbcde58b-e496-453a-9ec1-18007d6bf8b8","Type":"ContainerStarted","Data":"228a8ce4ab54c922925a8a41fc74d34ed9cdce7580084e6a9f0626d44dd2f69c"} Nov 04 01:03:02.222697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:02.222654 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d6x9n" podStartSLOduration=2.14000482 podStartE2EDuration="5.22264229s" totalImagesPullingTime="3.08263747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:02.221243938 +0000 UTC m=+14.244175985" watchObservedRunningTime="2026-11-04 01:03:02.22264229 +0000 UTC m=+14.245574294" Nov 04 01:03:06.843305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:06.843267 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" Nov 04 01:03:06.843771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:06.843424 4312 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:03:06.849213 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:06.849181 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" Nov 04 01:03:06.867394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:06.867320 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8wrmb" podStartSLOduration=5.752382817 podStartE2EDuration="9.867302915s" totalImagesPullingTime="4.114920098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:02.240513502 +0000 UTC m=+14.263445522" watchObservedRunningTime="2026-11-04 01:03:06.867302915 +0000 UTC m=+18.890234920" Nov 04 01:03:07.470040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:07.470011 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqgxv" Nov 04 01:03:11.204691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:11.204663 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f9sn6" Nov 04 01:03:12.873933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.873895 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4"] Nov 04 01:03:12.878400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.878376 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:12.893061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.893037 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 01:03:12.894306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.894275 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 01:03:12.911966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.911940 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4"] Nov 04 01:03:12.925886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.925861 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qm5kl\"" Nov 04 01:03:12.926058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.926038 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 04 01:03:12.926136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.926070 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 04 01:03:12.948233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.948205 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xr28v"] Nov 04 01:03:12.951942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.951920 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:12.952204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.952184 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:12.952292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.952226 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvjqn\" (UniqueName: \"kubernetes.io/projected/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-kube-api-access-dvjqn\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:12.952292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.952279 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:12.952566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.952537 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:12.967376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.967355 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t6wds\"" Nov 04 01:03:12.967376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.967366 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 01:03:12.967524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.967400 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 01:03:12.979576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.979557 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 01:03:12.981520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:12.981488 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 01:03:13.053137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053104 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053147 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053179 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-sys\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053253 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.053512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053414 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-textfile\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053482 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-root\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053514 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvjqn\" (UniqueName: \"kubernetes.io/projected/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-kube-api-access-dvjqn\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.053605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053570 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-wtmp\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053607 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.053904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053670 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-tls\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053710 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-metrics-client-ca\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053740 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9sgs\" (UniqueName: \"kubernetes.io/projected/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-kube-api-access-t9sgs\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.053904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.053808 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.054629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.054588 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.056276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.056255 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.056462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.056439 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.072348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.072306 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvjqn\" (UniqueName: \"kubernetes.io/projected/f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a-kube-api-access-dvjqn\") pod \"openshift-state-metrics-7bd7489957-wm6w4\" (UID: \"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.154722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154637 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154686 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154706 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-sys\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154726 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-textfile\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154788 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-sys\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154827 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-root\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154866 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-wtmp\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154896 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-tls\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154931 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-metrics-client-ca\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154942 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-root\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.154978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.154959 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sgs\" (UniqueName: \"kubernetes.io/projected/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-kube-api-access-t9sgs\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.155356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.155128 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-textfile\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.155356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.155258 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-wtmp\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.155356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.155320 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.155653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.155631 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-metrics-client-ca\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.157576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.157550 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-tls\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.158847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.158823 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.171950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.171925 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sgs\" (UniqueName: \"kubernetes.io/projected/c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf-kube-api-access-t9sgs\") pod \"node-exporter-xr28v\" (UID: \"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf\") " pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.196907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.196880 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" Nov 04 01:03:13.270647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.270621 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xr28v" Nov 04 01:03:13.285479 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:03:13.285447 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b2fc9f_84d5_4e6d_8acb_362c8ba7b6bf.slice/crio-01bf51bf59ea6fa08463f64821568731e0dfe2e585ed983f1b5f518e9f4a933e WatchSource:0}: Error finding container 01bf51bf59ea6fa08463f64821568731e0dfe2e585ed983f1b5f518e9f4a933e: Status 404 returned error can't find the container with id 01bf51bf59ea6fa08463f64821568731e0dfe2e585ed983f1b5f518e9f4a933e Nov 04 01:03:13.484883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:13.484815 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4"] Nov 04 01:03:13.488853 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:03:13.488824 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c1d676_2ee8_4722_b0c1_fa605ee6ec0a.slice/crio-d337c4ef251de9aafa3949461e17d34b1ae791cf160bab239a7c9f899ba210e8 WatchSource:0}: Error finding container d337c4ef251de9aafa3949461e17d34b1ae791cf160bab239a7c9f899ba210e8: Status 404 returned error can't find the container with id d337c4ef251de9aafa3949461e17d34b1ae791cf160bab239a7c9f899ba210e8 Nov 04 01:03:14.246390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.246342 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr28v" event={"ID":"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf","Type":"ContainerStarted","Data":"01bf51bf59ea6fa08463f64821568731e0dfe2e585ed983f1b5f518e9f4a933e"} Nov 04 01:03:14.248582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.248555 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" event={"ID":"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a","Type":"ContainerStarted","Data":"b3353e9aa83d100deb7971b369d4e6423d13ad9fd94c40e9a1386a8a47838da1"} Nov 04 01:03:14.248694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.248610 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" event={"ID":"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a","Type":"ContainerStarted","Data":"f8e4942f957c046ca358e425a7abe7c1cbb1d59681d46e5313612d853aea6b79"} Nov 04 01:03:14.248694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.248638 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" event={"ID":"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a","Type":"ContainerStarted","Data":"d337c4ef251de9aafa3949461e17d34b1ae791cf160bab239a7c9f899ba210e8"} Nov 04 01:03:14.886919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.886890 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh"] Nov 04 01:03:14.890937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.890914 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.907053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.906614 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 01:03:14.907053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.906749 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qrmhh\"" Nov 04 01:03:14.907053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.906829 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 01:03:14.907053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.906877 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 01:03:14.907053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.907001 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 01:03:14.907053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.907032 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9rmua6qjds08r\"" Nov 04 01:03:14.908167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.908148 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 01:03:14.909684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.909664 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh"] Nov 04 01:03:14.969294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969270 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da8a4c19-339b-45a6-9502-b02e0bc17482-metrics-client-ca\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969306 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-grpc-tls\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969355 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969445 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969497 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969520 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8a4c19-339b-45a6-9502-b02e0bc17482-tmp\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969557 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969594 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zd9\" (UniqueName: \"kubernetes.io/projected/da8a4c19-339b-45a6-9502-b02e0bc17482-kube-api-access-b8zd9\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:14.969679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:14.969610 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-tls\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.070936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.070903 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.070962 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.070994 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8a4c19-339b-45a6-9502-b02e0bc17482-tmp\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.071035 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.071052 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zd9\" (UniqueName: \"kubernetes.io/projected/da8a4c19-339b-45a6-9502-b02e0bc17482-kube-api-access-b8zd9\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.071097 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-tls\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.071114 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da8a4c19-339b-45a6-9502-b02e0bc17482-metrics-client-ca\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.071167 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-grpc-tls\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.071288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.071205 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.072850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.071698 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8a4c19-339b-45a6-9502-b02e0bc17482-tmp\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.072850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.072219 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da8a4c19-339b-45a6-9502-b02e0bc17482-metrics-client-ca\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.074205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.074182 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.074444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.074409 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.074841 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.074801 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.080377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.080325 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.080509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.080486 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-thanos-querier-tls\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.082674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.082647 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da8a4c19-339b-45a6-9502-b02e0bc17482-secret-grpc-tls\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.084832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.084811 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zd9\" (UniqueName: \"kubernetes.io/projected/da8a4c19-339b-45a6-9502-b02e0bc17482-kube-api-access-b8zd9\") pod \"thanos-querier-84f9cd5b47-fcwvh\" (UID: \"da8a4c19-339b-45a6-9502-b02e0bc17482\") " pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.237208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.237125 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:15.254680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.254638 4312 generic.go:356] "Generic (PLEG): container finished" podID="c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf" containerID="dc4eedac51b1d176c8b9747dc4766cd17678349d1914e5f4d47cc37000f8c4f4" exitCode=0 Nov 04 01:03:15.255007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.254755 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr28v" event={"ID":"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf","Type":"ContainerDied","Data":"dc4eedac51b1d176c8b9747dc4766cd17678349d1914e5f4d47cc37000f8c4f4"} Nov 04 01:03:15.262054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.261937 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" event={"ID":"f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a","Type":"ContainerStarted","Data":"76ade9fdb829bbbc8f8abbeb51221b96164b623475616599b141b28b409e3ec4"} Nov 04 01:03:15.300255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.300207 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wm6w4" podStartSLOduration=2.03792851 podStartE2EDuration="3.300187973s" totalImagesPullingTime="1.262259463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:15.2993017 +0000 UTC m=+27.322233738" watchObservedRunningTime="2026-11-04 01:03:15.300187973 +0000 UTC m=+27.323119983" Nov 04 01:03:15.522725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.522628 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh"] Nov 04 01:03:15.526977 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:03:15.526949 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8a4c19_339b_45a6_9502_b02e0bc17482.slice/crio-996c9b640bc334f86efbb9d5827733ebd72a9152ead05da94c92e8113ee5ce7c WatchSource:0}: Error finding container 996c9b640bc334f86efbb9d5827733ebd72a9152ead05da94c92e8113ee5ce7c: Status 404 returned error can't find the container with id 996c9b640bc334f86efbb9d5827733ebd72a9152ead05da94c92e8113ee5ce7c Nov 04 01:03:15.610780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:15.610750 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cs7zr" Nov 04 01:03:16.267231 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:16.267196 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" event={"ID":"da8a4c19-339b-45a6-9502-b02e0bc17482","Type":"ContainerStarted","Data":"996c9b640bc334f86efbb9d5827733ebd72a9152ead05da94c92e8113ee5ce7c"} Nov 04 01:03:16.269923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:16.269892 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr28v" event={"ID":"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf","Type":"ContainerStarted","Data":"2c485af0b303c8d1a3b2c43e590cd2113994fbce5e41dab9598c741e9d7ec3da"} Nov 04 01:03:16.270065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:16.269954 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr28v" event={"ID":"c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf","Type":"ContainerStarted","Data":"4edea63c65a4b20bd8e485ca8936ff24181171e36fc1cfd33be510e97c0f5bec"} Nov 04 01:03:17.611835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.611773 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xr28v" podStartSLOduration=3.85986913 podStartE2EDuration="5.611752679s" totalImagesPullingTime="1.674456076s" totalInitContainerRuntime="77.427473ms" isStatefulPod=true podCreationTimestamp="2026-11-04 01:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:16.293187507 +0000 UTC m=+28.316119513" watchObservedRunningTime="2026-11-04 01:03:17.611752679 +0000 UTC m=+29.634684687" Nov 04 01:03:17.612296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.612243 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl"] Nov 04 01:03:17.642678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.642648 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl"] Nov 04 01:03:17.642833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.642756 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:17.647514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.647485 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 01:03:17.647738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.647712 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d429t\"" Nov 04 01:03:17.739745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.739713 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tpsdw" Nov 04 01:03:17.794012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.793975 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/facd3002-3a6e-4511-bd92-ac51951ba75f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8nrdl\" (UID: \"facd3002-3a6e-4511-bd92-ac51951ba75f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:17.794192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.794099 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlzl\" (UniqueName: \"kubernetes.io/projected/facd3002-3a6e-4511-bd92-ac51951ba75f-kube-api-access-5wlzl\") pod \"monitoring-plugin-77f4f76594-8nrdl\" (UID: \"facd3002-3a6e-4511-bd92-ac51951ba75f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:17.895097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.895026 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlzl\" (UniqueName: \"kubernetes.io/projected/facd3002-3a6e-4511-bd92-ac51951ba75f-kube-api-access-5wlzl\") pod \"monitoring-plugin-77f4f76594-8nrdl\" (UID: \"facd3002-3a6e-4511-bd92-ac51951ba75f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:17.895097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.895090 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/facd3002-3a6e-4511-bd92-ac51951ba75f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8nrdl\" (UID: \"facd3002-3a6e-4511-bd92-ac51951ba75f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:17.898000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.897964 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/facd3002-3a6e-4511-bd92-ac51951ba75f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8nrdl\" (UID: \"facd3002-3a6e-4511-bd92-ac51951ba75f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:17.918076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.918045 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlzl\" (UniqueName: \"kubernetes.io/projected/facd3002-3a6e-4511-bd92-ac51951ba75f-kube-api-access-5wlzl\") pod \"monitoring-plugin-77f4f76594-8nrdl\" (UID: \"facd3002-3a6e-4511-bd92-ac51951ba75f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:17.969548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:17.969519 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:18.272069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:18.272047 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl"] Nov 04 01:03:18.342982 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:03:18.342947 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacd3002_3a6e_4511_bd92_ac51951ba75f.slice/crio-70533064a983783333266404cf4b9da5b72f7a4685b801ba480c3a5ad86964a6 WatchSource:0}: Error finding container 70533064a983783333266404cf4b9da5b72f7a4685b801ba480c3a5ad86964a6: Status 404 returned error can't find the container with id 70533064a983783333266404cf4b9da5b72f7a4685b801ba480c3a5ad86964a6 Nov 04 01:03:19.182916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.182888 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74fd556577-xc96m" Nov 04 01:03:19.284524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.284481 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" event={"ID":"da8a4c19-339b-45a6-9502-b02e0bc17482","Type":"ContainerStarted","Data":"921002086f896f0120bd05b687f50d3a55030d281ce17c34b26ab90943f74775"} Nov 04 01:03:19.284789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.284729 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" event={"ID":"da8a4c19-339b-45a6-9502-b02e0bc17482","Type":"ContainerStarted","Data":"d5b875287600ed35f35369c0150b7300e4208817c776776817cd14a81f0cac86"} Nov 04 01:03:19.286361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.286307 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" event={"ID":"facd3002-3a6e-4511-bd92-ac51951ba75f","Type":"ContainerStarted","Data":"70533064a983783333266404cf4b9da5b72f7a4685b801ba480c3a5ad86964a6"} Nov 04 01:03:19.291018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.288787 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gvtrm" event={"ID":"0e4ff47b-fe50-4203-bf00-1d332e02ff6d","Type":"ContainerStarted","Data":"df4b217aad46ab097038b89cf9c458142654ed0b95126f91020b1da47c67b459"} Nov 04 01:03:19.291018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.290390 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:03:19.294604 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.294584 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gvtrm" Nov 04 01:03:19.316879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:19.316658 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-gvtrm" podStartSLOduration=1.8057867239999998 podStartE2EDuration="22.316644051s" totalImagesPullingTime="20.510857327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:19.313799203 +0000 UTC m=+31.336731231" watchObservedRunningTime="2026-11-04 01:03:19.316644051 +0000 UTC m=+31.339576060" Nov 04 01:03:20.298060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:20.297953 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" event={"ID":"da8a4c19-339b-45a6-9502-b02e0bc17482","Type":"ContainerStarted","Data":"cbc428c5b779340e4de22b9a6770608ac97abf9292e557d0b38f56458707a334"} Nov 04 01:03:21.305069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:21.305034 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" event={"ID":"da8a4c19-339b-45a6-9502-b02e0bc17482","Type":"ContainerStarted","Data":"b9a0a2059691b701eb7f311d67a27fc594d6c381ab14de7bf4a2fa3c87dd83fc"} Nov 04 01:03:21.305546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:21.305123 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" event={"ID":"da8a4c19-339b-45a6-9502-b02e0bc17482","Type":"ContainerStarted","Data":"86fea9d128f594a5dda272c13556d1e7aa795d4a4dff0cd7f9bb02f29fbb517c"} Nov 04 01:03:21.306819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:21.306789 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" event={"ID":"facd3002-3a6e-4511-bd92-ac51951ba75f","Type":"ContainerStarted","Data":"957c2b5d6094ef264594812bd1998f6e2a260e583bdc89834769e418eaa16bfd"} Nov 04 01:03:21.307181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:21.307152 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:21.312615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:21.312586 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" Nov 04 01:03:21.325920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:21.325873 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8nrdl" podStartSLOduration=1.6642576930000001 podStartE2EDuration="4.325858031s" totalImagesPullingTime="2.661600338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:21.32457441 +0000 UTC m=+33.347506417" watchObservedRunningTime="2026-11-04 01:03:21.325858031 +0000 UTC m=+33.348790039" Nov 04 01:03:22.314714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:22.314670 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" event={"ID":"da8a4c19-339b-45a6-9502-b02e0bc17482","Type":"ContainerStarted","Data":"271b744001e1d346622b789ea457892e7040f62ffe5a6bb3cea2c90b0c10394c"} Nov 04 01:03:22.349637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:22.349581 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" podStartSLOduration=3.164002005 podStartE2EDuration="8.349565921s" totalImagesPullingTime="5.185563916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:22.347403648 +0000 UTC m=+34.370335675" watchObservedRunningTime="2026-11-04 01:03:22.349565921 +0000 UTC m=+34.372497906" Nov 04 01:03:23.319277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:23.319245 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:24.329413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:24.329382 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84f9cd5b47-fcwvh" Nov 04 01:03:29.457051 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:03:29.456971 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:03:56.700154 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.700124 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 01:03:56.746480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.746451 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 01:03:56.746632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.746572 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.750928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.750904 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 01:03:56.750928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.750908 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 01:03:56.751143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.750977 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 01:03:56.751143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.750988 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lkq85\"" Nov 04 01:03:56.751143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.750988 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 01:03:56.751143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.751000 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 01:03:56.751143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.751008 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 01:03:56.751143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.751024 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 01:03:56.752517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.752497 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 01:03:56.756991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.756974 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 01:03:56.782260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782237 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782261 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782283 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782301 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782383 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782458 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-web-config\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782504 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782532 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782573 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-config-out\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782645 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782681 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782708 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcn8\" (UniqueName: \"kubernetes.io/projected/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-kube-api-access-8wcn8\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.782831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.782773 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883405 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883439 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883457 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883475 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-web-config\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883496 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883524 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883560 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-config-out\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883580 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883605 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883628 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcn8\" (UniqueName: \"kubernetes.io/projected/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-kube-api-access-8wcn8\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883664 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883707 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.883948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.883735 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.884325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.884291 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.884536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.884506 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.884773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.884752 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.886277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.886258 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-config-out\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.886589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.886565 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.886971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.886669 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.887052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.887031 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.887315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.887290 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-web-config\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.887413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.887320 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.887754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.887732 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.887900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.887880 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.888191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.888170 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:56.896233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:56.896215 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcn8\" (UniqueName: \"kubernetes.io/projected/90c035f1-3ad5-4ce4-acf6-a84cf6ec433c-kube-api-access-8wcn8\") pod \"alertmanager-main-0\" (UID: \"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:57.155234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:57.155203 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 01:03:57.467590 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:57.467566 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 01:03:57.469550 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:03:57.469522 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c035f1_3ad5_4ce4_acf6_a84cf6ec433c.slice/crio-b1f34c8b86a3978d5a81b776ffb89b2824121391457a4b14238c9bef342beff0 WatchSource:0}: Error finding container b1f34c8b86a3978d5a81b776ffb89b2824121391457a4b14238c9bef342beff0: Status 404 returned error can't find the container with id b1f34c8b86a3978d5a81b776ffb89b2824121391457a4b14238c9bef342beff0 Nov 04 01:03:58.439563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:58.439530 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerStarted","Data":"b1f34c8b86a3978d5a81b776ffb89b2824121391457a4b14238c9bef342beff0"} Nov 04 01:03:59.443902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:59.443869 4312 generic.go:356] "Generic (PLEG): container finished" podID="90c035f1-3ad5-4ce4-acf6-a84cf6ec433c" containerID="1d8ce22bc0387483572f22376e073729f2194ab1757b9b74550db1227bfb3490" exitCode=0 Nov 04 01:03:59.444271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:03:59.443908 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerDied","Data":"1d8ce22bc0387483572f22376e073729f2194ab1757b9b74550db1227bfb3490"} Nov 04 01:03:59.456126 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:03:59.456097 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:04:02.459208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:04:02.459178 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerStarted","Data":"2f507da636828b8a64a9c73590a1954dcd5ec14733d953ab9d5184d507fdb098"} Nov 04 01:04:02.459511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:04:02.459249 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerStarted","Data":"7168598d64bba60e4a4b636aa2cb736cb5d66ccc8e90eb6ea4388ca2b45c5a6b"} Nov 04 01:04:02.459511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:04:02.459276 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerStarted","Data":"88245627e1e239b9791b4089fe717666508320555c1aef33c8ffc81549303b3e"} Nov 04 01:04:02.459511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:04:02.459299 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerStarted","Data":"f782aa50be0f00bafe24b4cd8cf78421b64cef577714bda4f09b9f234b389269"} Nov 04 01:04:03.466101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:04:03.466066 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerStarted","Data":"3617fc8c5b0cd1f827d5c48ef2aa7fe1d4516904bb58a59c92d142d17740e801"} Nov 04 01:04:03.466514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:04:03.466126 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90c035f1-3ad5-4ce4-acf6-a84cf6ec433c","Type":"ContainerStarted","Data":"c11cd75b95a09a50869d5093014cb0fc2ce56d40a38f40bd3762b70da78215dc"} Nov 04 01:04:03.501485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:04:03.501439 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.1047288650000002 podStartE2EDuration="7.501424863s" totalImagesPullingTime="4.383768787s" totalInitContainerRuntime="12.927211ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:03.498742428 +0000 UTC m=+75.521674435" watchObservedRunningTime="2026-11-04 01:04:03.501424863 +0000 UTC m=+75.524356945" Nov 04 01:04:29.456085 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:04:29.456051 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:04:59.455306 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:04:59.455224 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:05:29.455691 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:05:29.455648 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:05:59.456968 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:05:59.456932 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:06:24.722045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.722003 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2"] Nov 04 01:06:24.725351 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.725318 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.729792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.729757 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 01:06:24.730883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.730861 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 01:06:24.731114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.731087 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-klq4w\"" Nov 04 01:06:24.739691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.739656 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2"] Nov 04 01:06:24.841749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.841711 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk"] Nov 04 01:06:24.845060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.845036 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:24.850678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.850652 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-cm5pv\"" Nov 04 01:06:24.855447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.855421 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk"] Nov 04 01:06:24.880678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.880651 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.880809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.880687 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7sn\" (UniqueName: \"kubernetes.io/projected/add70cc1-e10e-40ce-b9ee-58a62618263f-kube-api-access-hq7sn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.880809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.880798 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.981226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981130 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nm6x\" (UniqueName: \"kubernetes.io/projected/097d0180-2726-45ad-a292-7e14591f8d5b-kube-api-access-9nm6x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:24.981414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981244 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:24.981414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981279 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.981414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981321 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.981414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981362 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:24.981414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981386 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7sn\" (UniqueName: \"kubernetes.io/projected/add70cc1-e10e-40ce-b9ee-58a62618263f-kube-api-access-hq7sn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.981678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981648 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.981751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.981724 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:24.993818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:24.993794 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7sn\" (UniqueName: \"kubernetes.io/projected/add70cc1-e10e-40ce-b9ee-58a62618263f-kube-api-access-hq7sn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:25.044953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.044919 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:25.082801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.082771 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:25.082972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.082820 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:25.082972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.082874 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nm6x\" (UniqueName: \"kubernetes.io/projected/097d0180-2726-45ad-a292-7e14591f8d5b-kube-api-access-9nm6x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:25.083225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.083201 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:25.083316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.083226 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:25.098968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.096379 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nm6x\" (UniqueName: \"kubernetes.io/projected/097d0180-2726-45ad-a292-7e14591f8d5b-kube-api-access-9nm6x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:25.161326 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.161294 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:25.228142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.228111 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj"] Nov 04 01:06:25.231794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.231738 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.236872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.236747 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 01:06:25.236872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.236838 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 01:06:25.236872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.236859 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 01:06:25.237725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.237700 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 01:06:25.237857 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.237797 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 01:06:25.238040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.238020 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 01:06:25.238131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.238115 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mdkcc\"" Nov 04 01:06:25.249287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.249260 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj"] Nov 04 01:06:25.320494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.320470 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2"] Nov 04 01:06:25.323184 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:06:25.323153 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd70cc1_e10e_40ce_b9ee_58a62618263f.slice/crio-2e33a28cca7e3b9d3b3da8b63ccd6d69e9b30cbbf95054b02b77cebe7463f39a WatchSource:0}: Error finding container 2e33a28cca7e3b9d3b3da8b63ccd6d69e9b30cbbf95054b02b77cebe7463f39a: Status 404 returned error can't find the container with id 2e33a28cca7e3b9d3b3da8b63ccd6d69e9b30cbbf95054b02b77cebe7463f39a Nov 04 01:06:25.386904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.386872 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd7adea-e355-4df7-9719-b604442d80c7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.387050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.386912 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.387050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.386941 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.387050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.387011 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.387050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.387046 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjbh\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-kube-api-access-mbjbh\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.387208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.387188 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.387250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.387238 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.438755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.438695 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk"] Nov 04 01:06:25.441268 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:06:25.441240 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097d0180_2726_45ad_a292_7e14591f8d5b.slice/crio-e427af37c7771a842e0e4ee7035757215b8317da232d580ffea4565e54b05f89 WatchSource:0}: Error finding container e427af37c7771a842e0e4ee7035757215b8317da232d580ffea4565e54b05f89: Status 404 returned error can't find the container with id e427af37c7771a842e0e4ee7035757215b8317da232d580ffea4565e54b05f89 Nov 04 01:06:25.487980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.487914 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.487980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.487967 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.488142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.488003 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd7adea-e355-4df7-9719-b604442d80c7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.488142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.488033 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.488142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.488071 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.488142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.488092 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.488142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.488114 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjbh\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-kube-api-access-mbjbh\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.488634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.488605 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.490552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.490523 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.490552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.490535 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd7adea-e355-4df7-9719-b604442d80c7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.490680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.490562 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.490680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.490576 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.500085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.500065 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.500347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.500318 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjbh\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-kube-api-access-mbjbh\") pod \"istiod-openshift-gateway-7876bfcb74-67gxj\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.545691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.545667 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:25.822919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.822891 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj"] Nov 04 01:06:25.824891 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:06:25.824866 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd7adea_e355_4df7_9719_b604442d80c7.slice/crio-8e39ee1870f28bf46104bc608fd4dacf445c67c5829413fdb1048367d721922c WatchSource:0}: Error finding container 8e39ee1870f28bf46104bc608fd4dacf445c67c5829413fdb1048367d721922c: Status 404 returned error can't find the container with id 8e39ee1870f28bf46104bc608fd4dacf445c67c5829413fdb1048367d721922c Nov 04 01:06:25.956074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.956044 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" event={"ID":"5bd7adea-e355-4df7-9719-b604442d80c7","Type":"ContainerStarted","Data":"8e39ee1870f28bf46104bc608fd4dacf445c67c5829413fdb1048367d721922c"} Nov 04 01:06:25.957180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.957154 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" event={"ID":"097d0180-2726-45ad-a292-7e14591f8d5b","Type":"ContainerStarted","Data":"e427af37c7771a842e0e4ee7035757215b8317da232d580ffea4565e54b05f89"} Nov 04 01:06:25.958210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:25.958186 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" event={"ID":"add70cc1-e10e-40ce-b9ee-58a62618263f","Type":"ContainerStarted","Data":"2e33a28cca7e3b9d3b3da8b63ccd6d69e9b30cbbf95054b02b77cebe7463f39a"} Nov 04 01:06:29.391375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:29.391315 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:06:29.391768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:29.391426 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:06:29.466674 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:06:29.466621 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:06:29.983049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:29.983009 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" event={"ID":"5bd7adea-e355-4df7-9719-b604442d80c7","Type":"ContainerStarted","Data":"fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4"} Nov 04 01:06:29.983817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:29.983788 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:30.990048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:30.989814 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:06:31.016705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:31.016643 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" podStartSLOduration=2.452909705 podStartE2EDuration="6.016629075s" totalImagesPullingTime="3.56371937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:30.011396702 +0000 UTC m=+222.034328725" watchObservedRunningTime="2026-11-04 01:06:31.016629075 +0000 UTC m=+223.039561081" Nov 04 01:06:32.997156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:32.997092 4312 generic.go:356] "Generic (PLEG): container finished" podID="097d0180-2726-45ad-a292-7e14591f8d5b" containerID="34ea9a64b311f2469c6da6d4590e5cbde1ffc2c597f8173d2b535d5ea35ffe0e" exitCode=0 Nov 04 01:06:32.997598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:32.997178 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" event={"ID":"097d0180-2726-45ad-a292-7e14591f8d5b","Type":"ContainerDied","Data":"34ea9a64b311f2469c6da6d4590e5cbde1ffc2c597f8173d2b535d5ea35ffe0e"} Nov 04 01:06:32.998801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:32.998778 4312 generic.go:356] "Generic (PLEG): container finished" podID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerID="46a7f1c79d425e4c8916d50bae876fd154f66a9f49ef5e45ba701400420ebb46" exitCode=0 Nov 04 01:06:32.998924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:32.998819 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" event={"ID":"add70cc1-e10e-40ce-b9ee-58a62618263f","Type":"ContainerDied","Data":"46a7f1c79d425e4c8916d50bae876fd154f66a9f49ef5e45ba701400420ebb46"} Nov 04 01:06:37.016662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:37.016625 4312 generic.go:356] "Generic (PLEG): container finished" podID="097d0180-2726-45ad-a292-7e14591f8d5b" containerID="4ef57799a453f2f576f1f4d058d5bc0251db76915946c3e4afb38f1bf3f0f675" exitCode=0 Nov 04 01:06:37.017104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:37.016712 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" event={"ID":"097d0180-2726-45ad-a292-7e14591f8d5b","Type":"ContainerDied","Data":"4ef57799a453f2f576f1f4d058d5bc0251db76915946c3e4afb38f1bf3f0f675"} Nov 04 01:06:37.018545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:37.018526 4312 generic.go:356] "Generic (PLEG): container finished" podID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerID="265292d3ee9408632719c58e617323fa8b8e15064e0cb836be907725dc163773" exitCode=0 Nov 04 01:06:37.018654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:37.018593 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" event={"ID":"add70cc1-e10e-40ce-b9ee-58a62618263f","Type":"ContainerDied","Data":"265292d3ee9408632719c58e617323fa8b8e15064e0cb836be907725dc163773"} Nov 04 01:06:46.057803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:46.057767 4312 generic.go:356] "Generic (PLEG): container finished" podID="097d0180-2726-45ad-a292-7e14591f8d5b" containerID="0b50fd7321cfa0b4e7ebb41ad51ac01007019fc9f5274dd4ebe9a294f1efb92f" exitCode=0 Nov 04 01:06:46.058281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:46.057817 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" event={"ID":"097d0180-2726-45ad-a292-7e14591f8d5b","Type":"ContainerDied","Data":"0b50fd7321cfa0b4e7ebb41ad51ac01007019fc9f5274dd4ebe9a294f1efb92f"} Nov 04 01:06:47.219517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.219486 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:47.282763 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.282709 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-util\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-util\") pod \"097d0180-2726-45ad-a292-7e14591f8d5b\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " Nov 04 01:06:47.282996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.282969 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-bundle\") pod \"097d0180-2726-45ad-a292-7e14591f8d5b\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " Nov 04 01:06:47.283161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.283018 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097d0180-2726-45ad-a292-7e14591f8d5b-kube-api-access-9nm6x\" (UniqueName: \"kubernetes.io/projected/097d0180-2726-45ad-a292-7e14591f8d5b-kube-api-access-9nm6x\") pod \"097d0180-2726-45ad-a292-7e14591f8d5b\" (UID: \"097d0180-2726-45ad-a292-7e14591f8d5b\") " Nov 04 01:06:47.283691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.283664 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-bundle" pod "097d0180-2726-45ad-a292-7e14591f8d5b" (UID: "097d0180-2726-45ad-a292-7e14591f8d5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:06:47.285218 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.285189 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097d0180-2726-45ad-a292-7e14591f8d5b-kube-api-access-9nm6x" pod "097d0180-2726-45ad-a292-7e14591f8d5b" (UID: "097d0180-2726-45ad-a292-7e14591f8d5b"). InnerVolumeSpecName "kube-api-access-9nm6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:06:47.287039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.287017 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-util" pod "097d0180-2726-45ad-a292-7e14591f8d5b" (UID: "097d0180-2726-45ad-a292-7e14591f8d5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:06:47.384391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.384362 4312 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-util\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:47.384391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.384386 4312 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/097d0180-2726-45ad-a292-7e14591f8d5b-bundle\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:47.384533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:47.384396 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nm6x\" (UniqueName: \"kubernetes.io/projected/097d0180-2726-45ad-a292-7e14591f8d5b-kube-api-access-9nm6x\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:48.066015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:48.065993 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" Nov 04 01:06:48.066178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:48.065989 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cm4qlk" event={"ID":"097d0180-2726-45ad-a292-7e14591f8d5b","Type":"ContainerDied","Data":"e427af37c7771a842e0e4ee7035757215b8317da232d580ffea4565e54b05f89"} Nov 04 01:06:48.066178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:48.066131 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e427af37c7771a842e0e4ee7035757215b8317da232d580ffea4565e54b05f89" Nov 04 01:06:48.068065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:48.068038 4312 generic.go:356] "Generic (PLEG): container finished" podID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerID="10dc36d54db49023d837a1cfb5a9826edee222da365d320002a5eafd90a41fe4" exitCode=0 Nov 04 01:06:48.068184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:48.068113 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" event={"ID":"add70cc1-e10e-40ce-b9ee-58a62618263f","Type":"ContainerDied","Data":"10dc36d54db49023d837a1cfb5a9826edee222da365d320002a5eafd90a41fe4"} Nov 04 01:06:49.214931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.214906 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:49.298228 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.298202 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add70cc1-e10e-40ce-b9ee-58a62618263f-kube-api-access-hq7sn\" (UniqueName: \"kubernetes.io/projected/add70cc1-e10e-40ce-b9ee-58a62618263f-kube-api-access-hq7sn\") pod \"add70cc1-e10e-40ce-b9ee-58a62618263f\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " Nov 04 01:06:49.298385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.298244 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-util\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-util\") pod \"add70cc1-e10e-40ce-b9ee-58a62618263f\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " Nov 04 01:06:49.298385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.298328 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-bundle\") pod \"add70cc1-e10e-40ce-b9ee-58a62618263f\" (UID: \"add70cc1-e10e-40ce-b9ee-58a62618263f\") " Nov 04 01:06:49.298669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.298646 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-bundle" pod "add70cc1-e10e-40ce-b9ee-58a62618263f" (UID: "add70cc1-e10e-40ce-b9ee-58a62618263f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:06:49.300263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.300239 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add70cc1-e10e-40ce-b9ee-58a62618263f-kube-api-access-hq7sn" pod "add70cc1-e10e-40ce-b9ee-58a62618263f" (UID: "add70cc1-e10e-40ce-b9ee-58a62618263f"). InnerVolumeSpecName "kube-api-access-hq7sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:06:49.302173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.302154 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-util" pod "add70cc1-e10e-40ce-b9ee-58a62618263f" (UID: "add70cc1-e10e-40ce-b9ee-58a62618263f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:06:49.399560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.399498 4312 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-bundle\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:49.399560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.399523 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq7sn\" (UniqueName: \"kubernetes.io/projected/add70cc1-e10e-40ce-b9ee-58a62618263f-kube-api-access-hq7sn\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:49.399560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:49.399533 4312 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add70cc1-e10e-40ce-b9ee-58a62618263f-util\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:50.079176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:50.079137 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" event={"ID":"add70cc1-e10e-40ce-b9ee-58a62618263f","Type":"ContainerDied","Data":"2e33a28cca7e3b9d3b3da8b63ccd6d69e9b30cbbf95054b02b77cebe7463f39a"} Nov 04 01:06:50.079176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:50.079164 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644429t2" Nov 04 01:06:50.079176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:06:50.079182 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e33a28cca7e3b9d3b3da8b63ccd6d69e9b30cbbf95054b02b77cebe7463f39a" Nov 04 01:06:59.455571 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:06:59.455537 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:07:06.347913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.347879 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl"] Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348207 4312 cpu_manager.go:414] "containerMap: removing container" podUID="097d0180-2726-45ad-a292-7e14591f8d5b" containerName="pull" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348220 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097d0180-2726-45ad-a292-7e14591f8d5b" containerName="pull" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348232 4312 cpu_manager.go:414] "containerMap: removing container" podUID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerName="util" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348238 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerName="util" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348245 4312 cpu_manager.go:414] "containerMap: removing container" podUID="097d0180-2726-45ad-a292-7e14591f8d5b" containerName="extract" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348252 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097d0180-2726-45ad-a292-7e14591f8d5b" containerName="extract" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348279 4312 cpu_manager.go:414] "containerMap: removing container" podUID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerName="pull" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348286 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerName="pull" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348296 4312 cpu_manager.go:414] "containerMap: removing container" podUID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerName="extract" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348301 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerName="extract" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348318 4312 cpu_manager.go:414] "containerMap: removing container" podUID="097d0180-2726-45ad-a292-7e14591f8d5b" containerName="util" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348325 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097d0180-2726-45ad-a292-7e14591f8d5b" containerName="util" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348396 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="097d0180-2726-45ad-a292-7e14591f8d5b" containerName="extract" Nov 04 01:07:06.349234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.348408 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="add70cc1-e10e-40ce-b9ee-58a62618263f" containerName="extract" Nov 04 01:07:06.350139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.350120 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.423764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.423729 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-manifests\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.423919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.423772 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-tmp\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.423919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.423798 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.423991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.423930 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gw4\" (UniqueName: \"kubernetes.io/projected/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-kube-api-access-f6gw4\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.423991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.423969 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.424095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.423990 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.437437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.437408 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 01:07:06.437584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.437437 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 01:07:06.437584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.437448 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 01:07:06.437584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.437437 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 01:07:06.437584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.437567 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 01:07:06.437793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.437754 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ptvlp\"" Nov 04 01:07:06.441429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.441408 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl"] Nov 04 01:07:06.525361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.525289 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gw4\" (UniqueName: \"kubernetes.io/projected/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-kube-api-access-f6gw4\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.525569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.525390 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.525569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.525422 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.525569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.525457 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-manifests\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.525569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.525490 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-tmp\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.525569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.525520 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.525996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.525970 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-manifests\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.527793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.527763 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-tmp\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.527913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.527861 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.528000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.527978 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.528000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.527991 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.537725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.537698 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gw4\" (UniqueName: \"kubernetes.io/projected/c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6-kube-api-access-f6gw4\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-56hwl\" (UID: \"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.663103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.663018 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:06.966920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:06.966897 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl"] Nov 04 01:07:06.968776 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:07:06.968748 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19dc4b5_9a3d_4e5a_ab01_52ef8293a5b6.slice/crio-d0031a54a7c3306966023547fd59a94058bb1bf1cebb9ff823037de9f89766bb WatchSource:0}: Error finding container d0031a54a7c3306966023547fd59a94058bb1bf1cebb9ff823037de9f89766bb: Status 404 returned error can't find the container with id d0031a54a7c3306966023547fd59a94058bb1bf1cebb9ff823037de9f89766bb Nov 04 01:07:07.140040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:07.140002 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" event={"ID":"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6","Type":"ContainerStarted","Data":"d0031a54a7c3306966023547fd59a94058bb1bf1cebb9ff823037de9f89766bb"} Nov 04 01:07:11.159553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:11.159514 4312 generic.go:356] "Generic (PLEG): container finished" podID="c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6" containerID="1a5abd1d616e6777522db26a2fc32e1f208d1061b62cc47dc2f16d1ddce85540" exitCode=0 Nov 04 01:07:11.159967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:11.159591 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" event={"ID":"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6","Type":"ContainerDied","Data":"1a5abd1d616e6777522db26a2fc32e1f208d1061b62cc47dc2f16d1ddce85540"} Nov 04 01:07:12.166188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:12.166141 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" event={"ID":"c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6","Type":"ContainerStarted","Data":"453fe227b0a72364a845494087c7179a5e55e893d370b5097cc91cd3a9ad9a98"} Nov 04 01:07:12.166615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:12.166319 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:12.213266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:12.213213 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" podStartSLOduration=2.047958808 podStartE2EDuration="6.213197333s" totalImagesPullingTime="3.675152539s" totalInitContainerRuntime="490.085986ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:12.212056759 +0000 UTC m=+264.234988810" watchObservedRunningTime="2026-11-04 01:07:12.213197333 +0000 UTC m=+264.236129342" Nov 04 01:07:23.172162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:23.172127 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" Nov 04 01:07:29.458314 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:07:29.458274 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:07:39.373878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.373847 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv"] Nov 04 01:07:39.377456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.377431 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.400029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.400002 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv"] Nov 04 01:07:39.400717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.400685 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-tmp\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.400835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.400745 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-tls-certs\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.400835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.400773 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczwj\" (UniqueName: \"kubernetes.io/projected/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-kube-api-access-jczwj\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.426937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.426913 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j5w8w\"" Nov 04 01:07:39.427098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.426949 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 01:07:39.427098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.426962 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 01:07:39.502195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.502135 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-tmp\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.502586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.502544 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-tls-certs\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.502769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.502751 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczwj\" (UniqueName: \"kubernetes.io/projected/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-kube-api-access-jczwj\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.506214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.506188 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-tmp\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.506551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.506529 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-tls-certs\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.522298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.522271 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczwj\" (UniqueName: \"kubernetes.io/projected/d6f638e8-b3b0-48dc-bb30-55f37633f3f7-kube-api-access-jczwj\") pod \"kube-auth-proxy-c488b8c84-pzcnv\" (UID: \"d6f638e8-b3b0-48dc-bb30-55f37633f3f7\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.690677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.690605 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" Nov 04 01:07:39.984237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:39.984169 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv"] Nov 04 01:07:39.987234 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:07:39.987181 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f638e8_b3b0_48dc_bb30_55f37633f3f7.slice/crio-9bc50ce2cebc87b4c804922829f9ee9185c97d8a14a19c247f6427f144e4f38d WatchSource:0}: Error finding container 9bc50ce2cebc87b4c804922829f9ee9185c97d8a14a19c247f6427f144e4f38d: Status 404 returned error can't find the container with id 9bc50ce2cebc87b4c804922829f9ee9185c97d8a14a19c247f6427f144e4f38d Nov 04 01:07:40.277762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:40.277728 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" event={"ID":"d6f638e8-b3b0-48dc-bb30-55f37633f3f7","Type":"ContainerStarted","Data":"9bc50ce2cebc87b4c804922829f9ee9185c97d8a14a19c247f6427f144e4f38d"} Nov 04 01:07:42.889010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.888969 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks"] Nov 04 01:07:42.891900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.891878 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:42.917325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.917301 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 01:07:42.917477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.917355 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 04 01:07:42.918370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.918329 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 04 01:07:42.918491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.918374 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 01:07:42.918491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.918425 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-czjxd\"" Nov 04 01:07:42.932451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.932426 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks"] Nov 04 01:07:42.933907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.933883 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12f17e0b-b49a-4e4b-b4a3-f65d75989758-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:42.934033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.933912 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12f17e0b-b49a-4e4b-b4a3-f65d75989758-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:42.934033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:42.933934 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lphn\" (UniqueName: \"kubernetes.io/projected/12f17e0b-b49a-4e4b-b4a3-f65d75989758-kube-api-access-5lphn\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.035073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.035050 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12f17e0b-b49a-4e4b-b4a3-f65d75989758-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.035174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.035080 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12f17e0b-b49a-4e4b-b4a3-f65d75989758-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.035174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.035101 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lphn\" (UniqueName: \"kubernetes.io/projected/12f17e0b-b49a-4e4b-b4a3-f65d75989758-kube-api-access-5lphn\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.035808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.035790 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12f17e0b-b49a-4e4b-b4a3-f65d75989758-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.037387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.037367 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12f17e0b-b49a-4e4b-b4a3-f65d75989758-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.047091 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.047068 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lphn\" (UniqueName: \"kubernetes.io/projected/12f17e0b-b49a-4e4b-b4a3-f65d75989758-kube-api-access-5lphn\") pod \"kuadrant-console-plugin-66c4fb575d-sdxks\" (UID: \"12f17e0b-b49a-4e4b-b4a3-f65d75989758\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.219463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.219381 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" Nov 04 01:07:43.296457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.296421 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" event={"ID":"d6f638e8-b3b0-48dc-bb30-55f37633f3f7","Type":"ContainerStarted","Data":"68afe7d79bd99512f4d295c4d0742eabbae414cd62a66b7577d09cade8ea9d61"} Nov 04 01:07:43.323790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.323729 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c488b8c84-pzcnv" podStartSLOduration=1.345267485 podStartE2EDuration="4.323713709s" totalImagesPullingTime="2.978446224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:43.321695002 +0000 UTC m=+295.344627036" watchObservedRunningTime="2026-11-04 01:07:43.323713709 +0000 UTC m=+295.346645715" Nov 04 01:07:43.502043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.501972 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks"] Nov 04 01:07:43.504800 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:07:43.504761 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12f17e0b_b49a_4e4b_b4a3_f65d75989758.slice/crio-6719cdfceb991bbfc9f764c4136a3f145207b2794ac7f700b7bd00f741d0b0a5 WatchSource:0}: Error finding container 6719cdfceb991bbfc9f764c4136a3f145207b2794ac7f700b7bd00f741d0b0a5: Status 404 returned error can't find the container with id 6719cdfceb991bbfc9f764c4136a3f145207b2794ac7f700b7bd00f741d0b0a5 Nov 04 01:07:43.805090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.805056 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dlvds"] Nov 04 01:07:43.807929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.807911 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" Nov 04 01:07:43.817842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.817818 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fkpdm\"" Nov 04 01:07:43.818588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.818565 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dlvds"] Nov 04 01:07:43.842258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.842229 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8x8\" (UniqueName: \"kubernetes.io/projected/2c14284f-583a-49a9-acb9-e3f53ac5b447-kube-api-access-wl8x8\") pod \"authorino-56c6c9ddd5-dlvds\" (UID: \"2c14284f-583a-49a9-acb9-e3f53ac5b447\") " pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" Nov 04 01:07:43.943279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.943241 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8x8\" (UniqueName: \"kubernetes.io/projected/2c14284f-583a-49a9-acb9-e3f53ac5b447-kube-api-access-wl8x8\") pod \"authorino-56c6c9ddd5-dlvds\" (UID: \"2c14284f-583a-49a9-acb9-e3f53ac5b447\") " pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" Nov 04 01:07:43.958726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:43.958698 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8x8\" (UniqueName: \"kubernetes.io/projected/2c14284f-583a-49a9-acb9-e3f53ac5b447-kube-api-access-wl8x8\") pod \"authorino-56c6c9ddd5-dlvds\" (UID: \"2c14284f-583a-49a9-acb9-e3f53ac5b447\") " pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" Nov 04 01:07:44.119086 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:44.119009 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" Nov 04 01:07:44.302487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:44.302453 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" event={"ID":"12f17e0b-b49a-4e4b-b4a3-f65d75989758","Type":"ContainerStarted","Data":"6719cdfceb991bbfc9f764c4136a3f145207b2794ac7f700b7bd00f741d0b0a5"} Nov 04 01:07:44.397715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:44.397573 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dlvds"] Nov 04 01:07:44.400262 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:07:44.400221 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c14284f_583a_49a9_acb9_e3f53ac5b447.slice/crio-c6c632d9ac3f7101020b783405e01ad608fe04950d69250e972b9e5f052d0e4e WatchSource:0}: Error finding container c6c632d9ac3f7101020b783405e01ad608fe04950d69250e972b9e5f052d0e4e: Status 404 returned error can't find the container with id c6c632d9ac3f7101020b783405e01ad608fe04950d69250e972b9e5f052d0e4e Nov 04 01:07:45.309159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:45.309123 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" event={"ID":"2c14284f-583a-49a9-acb9-e3f53ac5b447","Type":"ContainerStarted","Data":"c6c632d9ac3f7101020b783405e01ad608fe04950d69250e972b9e5f052d0e4e"} Nov 04 01:07:48.324083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:48.324042 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" event={"ID":"2c14284f-583a-49a9-acb9-e3f53ac5b447","Type":"ContainerStarted","Data":"fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273"} Nov 04 01:07:48.343459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:48.343413 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" podStartSLOduration=2.265935241 podStartE2EDuration="5.343401179s" totalImagesPullingTime="3.077465938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:48.341204339 +0000 UTC m=+300.364136371" watchObservedRunningTime="2026-11-04 01:07:48.343401179 +0000 UTC m=+300.366333188" Nov 04 01:07:50.335968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:50.335934 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" event={"ID":"12f17e0b-b49a-4e4b-b4a3-f65d75989758","Type":"ContainerStarted","Data":"91f9d315556e82fd93ddabe3688fde75f20b942514a8a7d51a161cc1849abb4f"} Nov 04 01:07:50.353881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:07:50.353833 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sdxks" podStartSLOduration=2.416279357 podStartE2EDuration="8.353819878s" totalImagesPullingTime="5.937540521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:50.352558237 +0000 UTC m=+302.375490267" watchObservedRunningTime="2026-11-04 01:07:50.353819878 +0000 UTC m=+302.376751885" Nov 04 01:07:59.455975 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:07:59.455888 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:08:29.456696 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:08:29.456649 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:08:35.342998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.342965 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cgt26"] Nov 04 01:08:35.346316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.346297 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.353862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.353841 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 01:08:35.354383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.354366 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 01:08:35.363167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.363138 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cgt26"] Nov 04 01:08:35.375039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.375001 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddtp\" (UniqueName: \"kubernetes.io/projected/b1513474-cbe8-4a4b-a267-9230887fe173-kube-api-access-mddtp\") pod \"model-serving-api-75bd89fdfc-cgt26\" (UID: \"b1513474-cbe8-4a4b-a267-9230887fe173\") " pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.375180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.375053 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1513474-cbe8-4a4b-a267-9230887fe173-tls-certs\") pod \"model-serving-api-75bd89fdfc-cgt26\" (UID: \"b1513474-cbe8-4a4b-a267-9230887fe173\") " pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.425240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.425208 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 04 01:08:35.425424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.425270 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-78c9g\"" Nov 04 01:08:35.476474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.476435 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddtp\" (UniqueName: \"kubernetes.io/projected/b1513474-cbe8-4a4b-a267-9230887fe173-kube-api-access-mddtp\") pod \"model-serving-api-75bd89fdfc-cgt26\" (UID: \"b1513474-cbe8-4a4b-a267-9230887fe173\") " pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.476474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.476477 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1513474-cbe8-4a4b-a267-9230887fe173-tls-certs\") pod \"model-serving-api-75bd89fdfc-cgt26\" (UID: \"b1513474-cbe8-4a4b-a267-9230887fe173\") " pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.478960 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.478936 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1513474-cbe8-4a4b-a267-9230887fe173-tls-certs\") pod \"model-serving-api-75bd89fdfc-cgt26\" (UID: \"b1513474-cbe8-4a4b-a267-9230887fe173\") " pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.489294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.489271 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddtp\" (UniqueName: \"kubernetes.io/projected/b1513474-cbe8-4a4b-a267-9230887fe173-kube-api-access-mddtp\") pod \"model-serving-api-75bd89fdfc-cgt26\" (UID: \"b1513474-cbe8-4a4b-a267-9230887fe173\") " pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.659524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.659438 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:35.932502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.932287 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cgt26"] Nov 04 01:08:35.935364 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:08:35.935317 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1513474_cbe8_4a4b_a267_9230887fe173.slice/crio-aa198fd206f088ce1991099692bf65fc2ff699c3d8c8bdb99461cc36fd37f007 WatchSource:0}: Error finding container aa198fd206f088ce1991099692bf65fc2ff699c3d8c8bdb99461cc36fd37f007: Status 404 returned error can't find the container with id aa198fd206f088ce1991099692bf65fc2ff699c3d8c8bdb99461cc36fd37f007 Nov 04 01:08:35.937900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.937877 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:08:35.959446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.959418 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp"] Nov 04 01:08:35.963683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.963659 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:35.976271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.976248 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp"] Nov 04 01:08:35.979531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.979509 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:35.979631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.979551 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:35.979677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.979627 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:35.979718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.979687 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:35.979766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.979718 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:35.979829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.979774 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spz2j\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-kube-api-access-spz2j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:35.979875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:35.979862 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.080563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.080522 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.080563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.080566 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.080817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.080599 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.080817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.080706 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.080817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.080743 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.080817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.080765 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spz2j\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-kube-api-access-spz2j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.080817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.080805 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.081756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.081728 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.083009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.082982 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.083107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.083011 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.083274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.083257 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.083451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.083433 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.092866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.092844 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spz2j\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-kube-api-access-spz2j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.093002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.092986 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-42sbp\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.277133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.277038 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:36.525511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.525471 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" event={"ID":"b1513474-cbe8-4a4b-a267-9230887fe173","Type":"ContainerStarted","Data":"aa198fd206f088ce1991099692bf65fc2ff699c3d8c8bdb99461cc36fd37f007"} Nov 04 01:08:36.557141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.557045 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp"] Nov 04 01:08:36.559313 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:08:36.559285 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa7d9ce_8af6_42ac_8eca_e80b5d3ec3fc.slice/crio-93fe92507b545d4c0f6c55f66b6160f61ff49c75537909918804bd07c43888b9 WatchSource:0}: Error finding container 93fe92507b545d4c0f6c55f66b6160f61ff49c75537909918804bd07c43888b9: Status 404 returned error can't find the container with id 93fe92507b545d4c0f6c55f66b6160f61ff49c75537909918804bd07c43888b9 Nov 04 01:08:36.561579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.561544 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:08:36.561665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:36.561618 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:08:37.532812 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:37.532772 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" event={"ID":"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc","Type":"ContainerStarted","Data":"b8c4188caf23f249b19a0c5394f02a7e41dc14b43c429f24d39eb8475ea23a89"} Nov 04 01:08:37.533300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:37.532873 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" event={"ID":"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc","Type":"ContainerStarted","Data":"93fe92507b545d4c0f6c55f66b6160f61ff49c75537909918804bd07c43888b9"} Nov 04 01:08:37.533300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:37.532903 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:37.535214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:37.534713 4312 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-42sbp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:08:37.535214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:37.534771 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:08:37.560930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:37.560883 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" podStartSLOduration=2.560871103 podStartE2EDuration="2.560871103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:37.56039575 +0000 UTC m=+349.583327784" watchObservedRunningTime="2026-11-04 01:08:37.560871103 +0000 UTC m=+349.583803109" Nov 04 01:08:38.538714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.538676 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" event={"ID":"b1513474-cbe8-4a4b-a267-9230887fe173","Type":"ContainerStarted","Data":"64730e7dd123d04c909279b49058625da3289a2f2915a360737acab61f80ea5c"} Nov 04 01:08:38.539131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.538762 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:38.539505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.539486 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:08:38.558227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.558182 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" podStartSLOduration=1.5512539379999999 podStartE2EDuration="3.558172115s" totalImagesPullingTime="2.006918177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:38.557069683 +0000 UTC m=+350.580001690" watchObservedRunningTime="2026-11-04 01:08:38.558172115 +0000 UTC m=+350.581104122" Nov 04 01:08:38.613206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.613177 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj"] Nov 04 01:08:38.613510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.613482 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" podUID="5bd7adea-e355-4df7-9719-b604442d80c7" containerName="discovery" containerID="cri-o://fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4" gracePeriod=30 Nov 04 01:08:38.854912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.854888 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:08:38.908404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908379 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-ca-configmap\") pod \"5bd7adea-e355-4df7-9719-b604442d80c7\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " Nov 04 01:08:38.908549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908431 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd7adea-e355-4df7-9719-b604442d80c7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd7adea-e355-4df7-9719-b604442d80c7-local-certs\") pod \"5bd7adea-e355-4df7-9719-b604442d80c7\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " Nov 04 01:08:38.908549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908482 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-istio-token\") pod \"5bd7adea-e355-4df7-9719-b604442d80c7\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " Nov 04 01:08:38.908549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908506 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-cacerts\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-cacerts\") pod \"5bd7adea-e355-4df7-9719-b604442d80c7\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " Nov 04 01:08:38.908549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908523 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-kubeconfig\") pod \"5bd7adea-e355-4df7-9719-b604442d80c7\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " Nov 04 01:08:38.908746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908555 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-dns-cert\") pod \"5bd7adea-e355-4df7-9719-b604442d80c7\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " Nov 04 01:08:38.908746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908611 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-kube-api-access-mbjbh\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-kube-api-access-mbjbh\") pod \"5bd7adea-e355-4df7-9719-b604442d80c7\" (UID: \"5bd7adea-e355-4df7-9719-b604442d80c7\") " Nov 04 01:08:38.908857 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908783 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-ca-configmap" pod "5bd7adea-e355-4df7-9719-b604442d80c7" (UID: "5bd7adea-e355-4df7-9719-b604442d80c7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:08:38.908923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.908905 4312 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-ca-configmap\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:08:38.910848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.910802 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd7adea-e355-4df7-9719-b604442d80c7-local-certs" pod "5bd7adea-e355-4df7-9719-b604442d80c7" (UID: "5bd7adea-e355-4df7-9719-b604442d80c7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:08:38.910972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.910888 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-cacerts" pod "5bd7adea-e355-4df7-9719-b604442d80c7" (UID: "5bd7adea-e355-4df7-9719-b604442d80c7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:08:38.911082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.911053 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-kubeconfig" pod "5bd7adea-e355-4df7-9719-b604442d80c7" (UID: "5bd7adea-e355-4df7-9719-b604442d80c7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:08:38.911209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.911187 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-istio-token" pod "5bd7adea-e355-4df7-9719-b604442d80c7" (UID: "5bd7adea-e355-4df7-9719-b604442d80c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:08:38.911248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.911189 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-kube-api-access-mbjbh" pod "5bd7adea-e355-4df7-9719-b604442d80c7" (UID: "5bd7adea-e355-4df7-9719-b604442d80c7"). InnerVolumeSpecName "kube-api-access-mbjbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:08:38.911365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:38.911328 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-dns-cert" pod "5bd7adea-e355-4df7-9719-b604442d80c7" (UID: "5bd7adea-e355-4df7-9719-b604442d80c7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:08:39.010134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.010100 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjbh\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-kube-api-access-mbjbh\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:08:39.010134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.010127 4312 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd7adea-e355-4df7-9719-b604442d80c7-local-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:08:39.010134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.010136 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd7adea-e355-4df7-9719-b604442d80c7-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:08:39.010359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.010147 4312 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-cacerts\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:08:39.010359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.010155 4312 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-kubeconfig\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:08:39.010359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.010164 4312 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bd7adea-e355-4df7-9719-b604442d80c7-istio-csr-dns-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:08:39.544301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.544268 4312 generic.go:356] "Generic (PLEG): container finished" podID="5bd7adea-e355-4df7-9719-b604442d80c7" containerID="fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4" exitCode=0 Nov 04 01:08:39.544787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.544314 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" Nov 04 01:08:39.544787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.544368 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" event={"ID":"5bd7adea-e355-4df7-9719-b604442d80c7","Type":"ContainerDied","Data":"fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4"} Nov 04 01:08:39.544787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.544415 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj" event={"ID":"5bd7adea-e355-4df7-9719-b604442d80c7","Type":"ContainerDied","Data":"8e39ee1870f28bf46104bc608fd4dacf445c67c5829413fdb1048367d721922c"} Nov 04 01:08:39.544787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.544433 4312 scope.go:122] "RemoveContainer" containerID="fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4" Nov 04 01:08:39.553449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.553433 4312 scope.go:122] "RemoveContainer" containerID="fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4" Nov 04 01:08:39.553718 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:08:39.553698 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4\": container with ID starting with fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4 not found: ID does not exist" containerID="fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4" Nov 04 01:08:39.553771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.553726 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4"} err="failed to get container status \"fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4\": rpc error: code = NotFound desc = could not find container \"fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4\": container with ID starting with fe06579fa52e1be30e6f0026d4a1a0f9fb579d50957d4e7b8a1bace2a3192cb4 not found: ID does not exist" Nov 04 01:08:39.566247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.566224 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj"] Nov 04 01:08:39.572161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:39.572141 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-67gxj"] Nov 04 01:08:40.106633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:40.106602 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd7adea-e355-4df7-9719-b604442d80c7" path="/var/lib/kubelet/pods/5bd7adea-e355-4df7-9719-b604442d80c7/volumes" Nov 04 01:08:49.549377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:08:49.549325 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-cgt26" Nov 04 01:08:59.456011 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:08:59.455970 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:09:02.495306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.495271 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2"] Nov 04 01:09:02.495793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.495625 4312 cpu_manager.go:414] "containerMap: removing container" podUID="5bd7adea-e355-4df7-9719-b604442d80c7" containerName="discovery" Nov 04 01:09:02.495793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.495639 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd7adea-e355-4df7-9719-b604442d80c7" containerName="discovery" Nov 04 01:09:02.495793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.495710 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd7adea-e355-4df7-9719-b604442d80c7" containerName="discovery" Nov 04 01:09:02.499255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.499238 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.507373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.507324 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:09:02.507580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.507561 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:09:02.507622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.507607 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rbkg2\"" Nov 04 01:09:02.507858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.507844 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:09:02.508005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.507994 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 01:09:02.516948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.516926 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2"] Nov 04 01:09:02.595492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.595462 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668ff6d4-ab75-40cc-9d1b-d60d908583ce-openshift-service-ca\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.595660 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.595498 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-metrics-certs\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.595660 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.595614 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-maas-api-tls\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.595660 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.595651 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnnb5\" (UniqueName: \"kubernetes.io/projected/668ff6d4-ab75-40cc-9d1b-d60d908583ce-kube-api-access-lnnb5\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.697005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.696970 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668ff6d4-ab75-40cc-9d1b-d60d908583ce-openshift-service-ca\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.697164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.697012 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-metrics-certs\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.697164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.697048 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-maas-api-tls\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.697164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.697083 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnnb5\" (UniqueName: \"kubernetes.io/projected/668ff6d4-ab75-40cc-9d1b-d60d908583ce-kube-api-access-lnnb5\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.697635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.697614 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668ff6d4-ab75-40cc-9d1b-d60d908583ce-openshift-service-ca\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.699550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.699528 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-maas-api-tls\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.699630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.699612 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-metrics-certs\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.713840 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.713807 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnnb5\" (UniqueName: \"kubernetes.io/projected/668ff6d4-ab75-40cc-9d1b-d60d908583ce-kube-api-access-lnnb5\") pod \"maas-api-674d648f7b-f2dg2\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:02.813632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:02.813603 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:03.088893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:03.088871 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2"] Nov 04 01:09:03.091021 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:09:03.090995 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668ff6d4_ab75_40cc_9d1b_d60d908583ce.slice/crio-f980f4da5b7dd7c2fc9a402d45cb596d092d929f6c82e653b429dcbe2e45da69 WatchSource:0}: Error finding container f980f4da5b7dd7c2fc9a402d45cb596d092d929f6c82e653b429dcbe2e45da69: Status 404 returned error can't find the container with id f980f4da5b7dd7c2fc9a402d45cb596d092d929f6c82e653b429dcbe2e45da69 Nov 04 01:09:03.641168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:03.641127 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" event={"ID":"668ff6d4-ab75-40cc-9d1b-d60d908583ce","Type":"ContainerStarted","Data":"f980f4da5b7dd7c2fc9a402d45cb596d092d929f6c82e653b429dcbe2e45da69"} Nov 04 01:09:06.657372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:06.657315 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" event={"ID":"668ff6d4-ab75-40cc-9d1b-d60d908583ce","Type":"ContainerStarted","Data":"c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58"} Nov 04 01:09:06.657784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:06.657474 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:06.678670 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:06.678626 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" podStartSLOduration=2.014545564 podStartE2EDuration="4.678612878s" totalImagesPullingTime="2.664067314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:06.677415727 +0000 UTC m=+378.700347753" watchObservedRunningTime="2026-11-04 01:09:06.678612878 +0000 UTC m=+378.701544884" Nov 04 01:09:12.666699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:12.666671 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:13.432342 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.432311 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7dxpg"] Nov 04 01:09:13.436477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.436456 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" Nov 04 01:09:13.443053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.443029 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7dxpg"] Nov 04 01:09:13.511864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.511829 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7dxpg"] Nov 04 01:09:13.512007 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:09:13.511991 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hd7kh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" podUID="93da6dd9-fd92-4217-9cb9-0bee00a9705d" Nov 04 01:09:13.544532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.544506 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b74xw"] Nov 04 01:09:13.548053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.548035 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.555450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.555425 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b74xw"] Nov 04 01:09:13.556634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.556617 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 01:09:13.589932 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.589908 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7kh\" (UniqueName: \"kubernetes.io/projected/93da6dd9-fd92-4217-9cb9-0bee00a9705d-kube-api-access-hd7kh\") pod \"authorino-6f6944bcdd-7dxpg\" (UID: \"93da6dd9-fd92-4217-9cb9-0bee00a9705d\") " pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" Nov 04 01:09:13.685885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.685816 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" Nov 04 01:09:13.690286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.690263 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-tls-cert\") pod \"authorino-6889cbd8c4-b74xw\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.690434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.690300 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdvc\" (UniqueName: \"kubernetes.io/projected/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-kube-api-access-jvdvc\") pod \"authorino-6889cbd8c4-b74xw\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.690481 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.690445 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7kh\" (UniqueName: \"kubernetes.io/projected/93da6dd9-fd92-4217-9cb9-0bee00a9705d-kube-api-access-hd7kh\") pod \"authorino-6f6944bcdd-7dxpg\" (UID: \"93da6dd9-fd92-4217-9cb9-0bee00a9705d\") " pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" Nov 04 01:09:13.692693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.692679 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" Nov 04 01:09:13.702753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.702733 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7kh\" (UniqueName: \"kubernetes.io/projected/93da6dd9-fd92-4217-9cb9-0bee00a9705d-kube-api-access-hd7kh\") pod \"authorino-6f6944bcdd-7dxpg\" (UID: \"93da6dd9-fd92-4217-9cb9-0bee00a9705d\") " pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" Nov 04 01:09:13.790949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.790922 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93da6dd9-fd92-4217-9cb9-0bee00a9705d-kube-api-access-hd7kh\" (UniqueName: \"kubernetes.io/projected/93da6dd9-fd92-4217-9cb9-0bee00a9705d-kube-api-access-hd7kh\") pod \"93da6dd9-fd92-4217-9cb9-0bee00a9705d\" (UID: \"93da6dd9-fd92-4217-9cb9-0bee00a9705d\") " Nov 04 01:09:13.791105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.791087 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-tls-cert\") pod \"authorino-6889cbd8c4-b74xw\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.791176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.791118 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdvc\" (UniqueName: \"kubernetes.io/projected/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-kube-api-access-jvdvc\") pod \"authorino-6889cbd8c4-b74xw\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.792997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.792974 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93da6dd9-fd92-4217-9cb9-0bee00a9705d-kube-api-access-hd7kh" pod "93da6dd9-fd92-4217-9cb9-0bee00a9705d" (UID: "93da6dd9-fd92-4217-9cb9-0bee00a9705d"). InnerVolumeSpecName "kube-api-access-hd7kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:09:13.793465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.793447 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-tls-cert\") pod \"authorino-6889cbd8c4-b74xw\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.802729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.802704 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdvc\" (UniqueName: \"kubernetes.io/projected/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-kube-api-access-jvdvc\") pod \"authorino-6889cbd8c4-b74xw\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.805234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.805214 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b74xw"] Nov 04 01:09:13.805416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.805404 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:13.834431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.834402 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b5bdbf74f-lkjbp"] Nov 04 01:09:13.838090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.838067 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:13.846015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.845991 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b5bdbf74f-lkjbp"] Nov 04 01:09:13.892402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.892377 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd7kh\" (UniqueName: \"kubernetes.io/projected/93da6dd9-fd92-4217-9cb9-0bee00a9705d-kube-api-access-hd7kh\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:13.993135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.993059 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhvp\" (UniqueName: \"kubernetes.io/projected/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-kube-api-access-5vhvp\") pod \"authorino-b5bdbf74f-lkjbp\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:13.993135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:13.993105 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-tls-cert\") pod \"authorino-b5bdbf74f-lkjbp\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:14.078843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.078821 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b74xw"] Nov 04 01:09:14.081251 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:09:14.081218 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod470f03a1_9cef_46b5_bda3_9dae6d8ef3cf.slice/crio-e13cb250613647c56fe77430f65063b446b781c7af9ab2b2d84bccb25911cf1f WatchSource:0}: Error finding container e13cb250613647c56fe77430f65063b446b781c7af9ab2b2d84bccb25911cf1f: Status 404 returned error can't find the container with id e13cb250613647c56fe77430f65063b446b781c7af9ab2b2d84bccb25911cf1f Nov 04 01:09:14.093843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.093821 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhvp\" (UniqueName: \"kubernetes.io/projected/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-kube-api-access-5vhvp\") pod \"authorino-b5bdbf74f-lkjbp\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:14.093926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.093855 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-tls-cert\") pod \"authorino-b5bdbf74f-lkjbp\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:14.096160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.096137 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-tls-cert\") pod \"authorino-b5bdbf74f-lkjbp\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:14.107217 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.107196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhvp\" (UniqueName: \"kubernetes.io/projected/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-kube-api-access-5vhvp\") pod \"authorino-b5bdbf74f-lkjbp\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:14.153064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.153042 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:09:14.422097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.422072 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b5bdbf74f-lkjbp"] Nov 04 01:09:14.424422 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:09:14.424394 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ed660ad_f2c7_4a1c_adae_773cd5f0a6cc.slice/crio-401fbe59306ddbca552c87a2088afea85a377ca7ff918bf9ec8ea63cdb833205 WatchSource:0}: Error finding container 401fbe59306ddbca552c87a2088afea85a377ca7ff918bf9ec8ea63cdb833205: Status 404 returned error can't find the container with id 401fbe59306ddbca552c87a2088afea85a377ca7ff918bf9ec8ea63cdb833205 Nov 04 01:09:14.690761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.690682 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" event={"ID":"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc","Type":"ContainerStarted","Data":"401fbe59306ddbca552c87a2088afea85a377ca7ff918bf9ec8ea63cdb833205"} Nov 04 01:09:14.691838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.691822 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7dxpg" Nov 04 01:09:14.691910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.691821 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" event={"ID":"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf","Type":"ContainerStarted","Data":"e13cb250613647c56fe77430f65063b446b781c7af9ab2b2d84bccb25911cf1f"} Nov 04 01:09:14.722679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.722652 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7dxpg"] Nov 04 01:09:14.728764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:14.728743 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7dxpg"] Nov 04 01:09:15.700459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:15.700371 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" event={"ID":"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf","Type":"ContainerStarted","Data":"7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5"} Nov 04 01:09:15.700887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:15.700485 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" podUID="470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" containerName="authorino" containerID="cri-o://7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5" gracePeriod=30 Nov 04 01:09:15.703190 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:15.702598 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" event={"ID":"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc","Type":"ContainerStarted","Data":"5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3"} Nov 04 01:09:15.719587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:15.719522 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" podStartSLOduration=1.632607008 podStartE2EDuration="2.719504129s" totalImagesPullingTime="1.086897121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:15.717451737 +0000 UTC m=+387.740383745" watchObservedRunningTime="2026-11-04 01:09:15.719504129 +0000 UTC m=+387.742436138" Nov 04 01:09:15.739648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:15.739600 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" podStartSLOduration=1.730101018 podStartE2EDuration="2.739586183s" totalImagesPullingTime="1.009485165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:15.73771682 +0000 UTC m=+387.760648836" watchObservedRunningTime="2026-11-04 01:09:15.739586183 +0000 UTC m=+387.762518361" Nov 04 01:09:15.777852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:15.777812 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dlvds"] Nov 04 01:09:15.778552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:15.778281 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" podUID="2c14284f-583a-49a9-acb9-e3f53ac5b447" containerName="authorino" containerID="cri-o://fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273" gracePeriod=30 Nov 04 01:09:16.039367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.033653 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:16.051543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.051515 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" Nov 04 01:09:16.106463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.106422 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93da6dd9-fd92-4217-9cb9-0bee00a9705d" path="/var/lib/kubelet/pods/93da6dd9-fd92-4217-9cb9-0bee00a9705d/volumes" Nov 04 01:09:16.216621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.216587 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c14284f-583a-49a9-acb9-e3f53ac5b447-kube-api-access-wl8x8\" (UniqueName: \"kubernetes.io/projected/2c14284f-583a-49a9-acb9-e3f53ac5b447-kube-api-access-wl8x8\") pod \"2c14284f-583a-49a9-acb9-e3f53ac5b447\" (UID: \"2c14284f-583a-49a9-acb9-e3f53ac5b447\") " Nov 04 01:09:16.216789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.216653 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-kube-api-access-jvdvc\" (UniqueName: \"kubernetes.io/projected/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-kube-api-access-jvdvc\") pod \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " Nov 04 01:09:16.216789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.216696 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-tls-cert\" (UniqueName: \"kubernetes.io/secret/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-tls-cert\") pod \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\" (UID: \"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf\") " Nov 04 01:09:16.218842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.218807 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-kube-api-access-jvdvc" pod "470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" (UID: "470f03a1-9cef-46b5-bda3-9dae6d8ef3cf"). InnerVolumeSpecName "kube-api-access-jvdvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:09:16.219087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.219062 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c14284f-583a-49a9-acb9-e3f53ac5b447-kube-api-access-wl8x8" pod "2c14284f-583a-49a9-acb9-e3f53ac5b447" (UID: "2c14284f-583a-49a9-acb9-e3f53ac5b447"). InnerVolumeSpecName "kube-api-access-wl8x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:09:16.226686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.226663 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-tls-cert" pod "470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" (UID: "470f03a1-9cef-46b5-bda3-9dae6d8ef3cf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:09:16.318046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.318020 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8x8\" (UniqueName: \"kubernetes.io/projected/2c14284f-583a-49a9-acb9-e3f53ac5b447-kube-api-access-wl8x8\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:16.318046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.318044 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvdvc\" (UniqueName: \"kubernetes.io/projected/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-kube-api-access-jvdvc\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:16.318209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.318054 4312 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf-tls-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:16.708207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.708178 4312 generic.go:356] "Generic (PLEG): container finished" podID="2c14284f-583a-49a9-acb9-e3f53ac5b447" containerID="fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273" exitCode=0 Nov 04 01:09:16.708567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.708220 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" Nov 04 01:09:16.708567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.708262 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" event={"ID":"2c14284f-583a-49a9-acb9-e3f53ac5b447","Type":"ContainerDied","Data":"fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273"} Nov 04 01:09:16.708567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.708301 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dlvds" event={"ID":"2c14284f-583a-49a9-acb9-e3f53ac5b447","Type":"ContainerDied","Data":"c6c632d9ac3f7101020b783405e01ad608fe04950d69250e972b9e5f052d0e4e"} Nov 04 01:09:16.708567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.708324 4312 scope.go:122] "RemoveContainer" containerID="fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273" Nov 04 01:09:16.710472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.710446 4312 generic.go:356] "Generic (PLEG): container finished" podID="470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" containerID="7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5" exitCode=0 Nov 04 01:09:16.710594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.710477 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" event={"ID":"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf","Type":"ContainerDied","Data":"7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5"} Nov 04 01:09:16.710594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.710494 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" Nov 04 01:09:16.710594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.710520 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b74xw" event={"ID":"470f03a1-9cef-46b5-bda3-9dae6d8ef3cf","Type":"ContainerDied","Data":"e13cb250613647c56fe77430f65063b446b781c7af9ab2b2d84bccb25911cf1f"} Nov 04 01:09:16.718060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.718040 4312 scope.go:122] "RemoveContainer" containerID="fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273" Nov 04 01:09:16.718453 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:09:16.718328 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273\": container with ID starting with fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273 not found: ID does not exist" containerID="fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273" Nov 04 01:09:16.718546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.718461 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273"} err="failed to get container status \"fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273\": rpc error: code = NotFound desc = could not find container \"fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273\": container with ID starting with fb016f67a07e3da147e59d959039c0ed3c0ec4c091ac91a85b5a8bc99099e273 not found: ID does not exist" Nov 04 01:09:16.718546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.718477 4312 scope.go:122] "RemoveContainer" containerID="7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5" Nov 04 01:09:16.726158 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.726134 4312 scope.go:122] "RemoveContainer" containerID="7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5" Nov 04 01:09:16.726418 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:09:16.726400 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5\": container with ID starting with 7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5 not found: ID does not exist" containerID="7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5" Nov 04 01:09:16.726482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.726424 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5"} err="failed to get container status \"7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5\": rpc error: code = NotFound desc = could not find container \"7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5\": container with ID starting with 7c4a0a3fb9f8bb63e12c95486f3081f1c4b1d2d97fedb9a987b6d2b9037498a5 not found: ID does not exist" Nov 04 01:09:16.731490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.731471 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dlvds"] Nov 04 01:09:16.735860 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.735841 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dlvds"] Nov 04 01:09:16.746435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.746413 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b74xw"] Nov 04 01:09:16.750310 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:16.750293 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b74xw"] Nov 04 01:09:18.106576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:18.106546 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c14284f-583a-49a9-acb9-e3f53ac5b447" path="/var/lib/kubelet/pods/2c14284f-583a-49a9-acb9-e3f53ac5b447/volumes" Nov 04 01:09:18.107070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:18.106875 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" path="/var/lib/kubelet/pods/470f03a1-9cef-46b5-bda3-9dae6d8ef3cf/volumes" Nov 04 01:09:25.320127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.320086 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2"] Nov 04 01:09:25.320613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.320372 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" podUID="668ff6d4-ab75-40cc-9d1b-d60d908583ce" containerName="maas-api" containerID="cri-o://c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58" gracePeriod=30 Nov 04 01:09:25.565740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.565718 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:25.589923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.589862 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-maas-api-tls\") pod \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " Nov 04 01:09:25.590049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.589990 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668ff6d4-ab75-40cc-9d1b-d60d908583ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668ff6d4-ab75-40cc-9d1b-d60d908583ce-openshift-service-ca\") pod \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " Nov 04 01:09:25.590049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.590016 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-metrics-certs\") pod \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " Nov 04 01:09:25.590160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.590059 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668ff6d4-ab75-40cc-9d1b-d60d908583ce-kube-api-access-lnnb5\" (UniqueName: \"kubernetes.io/projected/668ff6d4-ab75-40cc-9d1b-d60d908583ce-kube-api-access-lnnb5\") pod \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\" (UID: \"668ff6d4-ab75-40cc-9d1b-d60d908583ce\") " Nov 04 01:09:25.590427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.590387 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668ff6d4-ab75-40cc-9d1b-d60d908583ce-openshift-service-ca" pod "668ff6d4-ab75-40cc-9d1b-d60d908583ce" (UID: "668ff6d4-ab75-40cc-9d1b-d60d908583ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:09:25.592093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.592063 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-maas-api-tls" pod "668ff6d4-ab75-40cc-9d1b-d60d908583ce" (UID: "668ff6d4-ab75-40cc-9d1b-d60d908583ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:09:25.592093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.592086 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-metrics-certs" pod "668ff6d4-ab75-40cc-9d1b-d60d908583ce" (UID: "668ff6d4-ab75-40cc-9d1b-d60d908583ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:09:25.592410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.592383 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668ff6d4-ab75-40cc-9d1b-d60d908583ce-kube-api-access-lnnb5" pod "668ff6d4-ab75-40cc-9d1b-d60d908583ce" (UID: "668ff6d4-ab75-40cc-9d1b-d60d908583ce"). InnerVolumeSpecName "kube-api-access-lnnb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:09:25.691299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.691270 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668ff6d4-ab75-40cc-9d1b-d60d908583ce-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:25.691299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.691300 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:25.691506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.691312 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnnb5\" (UniqueName: \"kubernetes.io/projected/668ff6d4-ab75-40cc-9d1b-d60d908583ce-kube-api-access-lnnb5\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:25.691506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.691323 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668ff6d4-ab75-40cc-9d1b-d60d908583ce-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:25.749963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.749927 4312 generic.go:356] "Generic (PLEG): container finished" podID="668ff6d4-ab75-40cc-9d1b-d60d908583ce" containerID="c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58" exitCode=0 Nov 04 01:09:25.750120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.749968 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" Nov 04 01:09:25.750120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.749993 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" event={"ID":"668ff6d4-ab75-40cc-9d1b-d60d908583ce","Type":"ContainerDied","Data":"c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58"} Nov 04 01:09:25.750120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.750041 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2" event={"ID":"668ff6d4-ab75-40cc-9d1b-d60d908583ce","Type":"ContainerDied","Data":"f980f4da5b7dd7c2fc9a402d45cb596d092d929f6c82e653b429dcbe2e45da69"} Nov 04 01:09:25.750120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.750064 4312 scope.go:122] "RemoveContainer" containerID="c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58" Nov 04 01:09:25.764501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.764479 4312 scope.go:122] "RemoveContainer" containerID="c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58" Nov 04 01:09:25.764779 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:09:25.764759 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58\": container with ID starting with c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58 not found: ID does not exist" containerID="c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58" Nov 04 01:09:25.764859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.764791 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58"} err="failed to get container status \"c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58\": rpc error: code = NotFound desc = could not find container \"c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58\": container with ID starting with c21a32282e960cb580b214ced6a464ba89f2d0c6d1321c66039e1ee951063a58 not found: ID does not exist" Nov 04 01:09:25.775155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.775133 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2"] Nov 04 01:09:25.781074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:25.781050 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-f2dg2"] Nov 04 01:09:26.105831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:09:26.105797 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668ff6d4-ab75-40cc-9d1b-d60d908583ce" path="/var/lib/kubelet/pods/668ff6d4-ab75-40cc-9d1b-d60d908583ce/volumes" Nov 04 01:09:29.456812 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:09:29.456731 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:09:59.455951 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:09:59.455910 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:10:26.900672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.900638 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8"] Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.900953 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2c14284f-583a-49a9-acb9-e3f53ac5b447" containerName="authorino" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.900963 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c14284f-583a-49a9-acb9-e3f53ac5b447" containerName="authorino" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.900971 4312 cpu_manager.go:414] "containerMap: removing container" podUID="470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" containerName="authorino" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.900976 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" containerName="authorino" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.900989 4312 cpu_manager.go:414] "containerMap: removing container" podUID="668ff6d4-ab75-40cc-9d1b-d60d908583ce" containerName="maas-api" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.900994 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668ff6d4-ab75-40cc-9d1b-d60d908583ce" containerName="maas-api" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.901056 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="668ff6d4-ab75-40cc-9d1b-d60d908583ce" containerName="maas-api" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.901069 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c14284f-583a-49a9-acb9-e3f53ac5b447" containerName="authorino" Nov 04 01:10:26.901138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.901075 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="470f03a1-9cef-46b5-bda3-9dae6d8ef3cf" containerName="authorino" Nov 04 01:10:26.904125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.904103 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:26.910996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.910447 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kbz2n\"" Nov 04 01:10:26.914939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.914901 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 01:10:26.915720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.915303 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 01:10:26.915720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.915269 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 01:10:26.920669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:26.920629 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8"] Nov 04 01:10:27.078053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.078016 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.078053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.078058 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.078251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.078165 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.078251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.078214 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.078316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.078287 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc62\" (UniqueName: \"kubernetes.io/projected/44760365-91dc-4a74-b176-231e25a8df60-kube-api-access-qdc62\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.078368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.078328 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44760365-91dc-4a74-b176-231e25a8df60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.179572 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179490 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.179572 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179532 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc62\" (UniqueName: \"kubernetes.io/projected/44760365-91dc-4a74-b176-231e25a8df60-kube-api-access-qdc62\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.179572 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179558 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44760365-91dc-4a74-b176-231e25a8df60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.179854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179594 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.179854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179614 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.179854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179676 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.180018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179945 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.180018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.179991 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.180111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.180071 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.181836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.181817 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44760365-91dc-4a74-b176-231e25a8df60-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.182012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.181996 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44760365-91dc-4a74-b176-231e25a8df60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.211802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.211768 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc62\" (UniqueName: \"kubernetes.io/projected/44760365-91dc-4a74-b176-231e25a8df60-kube-api-access-qdc62\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8\" (UID: \"44760365-91dc-4a74-b176-231e25a8df60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.217524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.217504 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:27.495824 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.495796 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8"] Nov 04 01:10:27.498393 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:10:27.498364 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44760365_91dc_4a74_b176_231e25a8df60.slice/crio-d9793b64c494abd48a209923ce7175a38f9dbd9782a53e5031a85ee66ad75abd WatchSource:0}: Error finding container d9793b64c494abd48a209923ce7175a38f9dbd9782a53e5031a85ee66ad75abd: Status 404 returned error can't find the container with id d9793b64c494abd48a209923ce7175a38f9dbd9782a53e5031a85ee66ad75abd Nov 04 01:10:27.873607 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.873577 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz"] Nov 04 01:10:27.878727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.878701 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:27.886215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.886192 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 04 01:10:27.889622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.889595 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz"] Nov 04 01:10:27.985959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.985922 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:27.986455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.985966 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f659cf1c-b69f-436b-8bf0-9bbddd61383d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:27.986455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.986009 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmjt\" (UniqueName: \"kubernetes.io/projected/f659cf1c-b69f-436b-8bf0-9bbddd61383d-kube-api-access-tmmjt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:27.986455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.986041 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:27.986455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.986072 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:27.986455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:27.986134 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.003476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.003365 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" event={"ID":"44760365-91dc-4a74-b176-231e25a8df60","Type":"ContainerStarted","Data":"d9793b64c494abd48a209923ce7175a38f9dbd9782a53e5031a85ee66ad75abd"} Nov 04 01:10:28.087600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087087 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.087600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087194 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.087600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087239 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.087600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087266 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f659cf1c-b69f-436b-8bf0-9bbddd61383d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.087600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087313 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmjt\" (UniqueName: \"kubernetes.io/projected/f659cf1c-b69f-436b-8bf0-9bbddd61383d-kube-api-access-tmmjt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.087600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087369 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.088037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087703 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.088037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087735 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.088037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.087868 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.090035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.090016 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f659cf1c-b69f-436b-8bf0-9bbddd61383d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.090323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.090302 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f659cf1c-b69f-436b-8bf0-9bbddd61383d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.103769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.100075 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmjt\" (UniqueName: \"kubernetes.io/projected/f659cf1c-b69f-436b-8bf0-9bbddd61383d-kube-api-access-tmmjt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz\" (UID: \"f659cf1c-b69f-436b-8bf0-9bbddd61383d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.196585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.196495 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:28.489576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:28.489545 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz"] Nov 04 01:10:28.492375 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:10:28.492325 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf659cf1c_b69f_436b_8bf0_9bbddd61383d.slice/crio-b0a447755567025ae3109738e64cb534eadb6617fdb4f2425f19519912dd6b1b WatchSource:0}: Error finding container b0a447755567025ae3109738e64cb534eadb6617fdb4f2425f19519912dd6b1b: Status 404 returned error can't find the container with id b0a447755567025ae3109738e64cb534eadb6617fdb4f2425f19519912dd6b1b Nov 04 01:10:29.010968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:29.010931 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" event={"ID":"f659cf1c-b69f-436b-8bf0-9bbddd61383d","Type":"ContainerStarted","Data":"b0a447755567025ae3109738e64cb534eadb6617fdb4f2425f19519912dd6b1b"} Nov 04 01:10:29.458399 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:10:29.458359 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:10:30.016349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.016298 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" event={"ID":"44760365-91dc-4a74-b176-231e25a8df60","Type":"ContainerStarted","Data":"78640f2b49a25f0f3a8cf82172a49b1dd4e57635a39556e5076b334e7ca4ed9e"} Nov 04 01:10:30.017762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.017737 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" event={"ID":"f659cf1c-b69f-436b-8bf0-9bbddd61383d","Type":"ContainerStarted","Data":"6396480b78c3a5df8917e99ddfe42302484394b9fcd0d2a234341227089f92b4"} Nov 04 01:10:30.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.037945 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" podStartSLOduration=2.441504254 podStartE2EDuration="4.03793068s" totalImagesPullingTime="1.596426426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:30.037062372 +0000 UTC m=+462.059994390" watchObservedRunningTime="2026-11-04 01:10:30.03793068 +0000 UTC m=+462.060862687" Nov 04 01:10:30.063534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.059706 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podStartSLOduration=1.9791594350000001 podStartE2EDuration="3.05969028s" totalImagesPullingTime="1.080530845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:30.057408629 +0000 UTC m=+462.080340637" watchObservedRunningTime="2026-11-04 01:10:30.05969028 +0000 UTC m=+462.082622287" Nov 04 01:10:30.196998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.196964 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:30.217601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.217568 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:30.277851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.277820 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9"] Nov 04 01:10:30.281024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.281006 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.290068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.290047 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 01:10:30.291764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.291740 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9"] Nov 04 01:10:30.408428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.408391 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.408598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.408556 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.408676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.408614 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nm8\" (UniqueName: \"kubernetes.io/projected/a7f1bc94-14c3-4783-9aad-2efc85a264e1-kube-api-access-v6nm8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.408676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.408655 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f1bc94-14c3-4783-9aad-2efc85a264e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.408747 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.408731 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.408798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.408759 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.509777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.509744 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.509777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.509778 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.509947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.509793 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.509947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.509844 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.509947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.509866 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nm8\" (UniqueName: \"kubernetes.io/projected/a7f1bc94-14c3-4783-9aad-2efc85a264e1-kube-api-access-v6nm8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.509947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.509887 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f1bc94-14c3-4783-9aad-2efc85a264e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.510234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.510212 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.510290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.510212 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.510290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.510280 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.512066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.512041 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f1bc94-14c3-4783-9aad-2efc85a264e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.512303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.512284 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f1bc94-14c3-4783-9aad-2efc85a264e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.521753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.521735 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nm8\" (UniqueName: \"kubernetes.io/projected/a7f1bc94-14c3-4783-9aad-2efc85a264e1-kube-api-access-v6nm8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9\" (UID: \"a7f1bc94-14c3-4783-9aad-2efc85a264e1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.594818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.594763 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:30.867035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:30.867012 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9"] Nov 04 01:10:30.868995 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:10:30.868963 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f1bc94_14c3_4783_9aad_2efc85a264e1.slice/crio-5aa77a95ed36b51082d86d2e2b3d02dc448ede1d9f4ddc3f05f7ad774b2eaf43 WatchSource:0}: Error finding container 5aa77a95ed36b51082d86d2e2b3d02dc448ede1d9f4ddc3f05f7ad774b2eaf43: Status 404 returned error can't find the container with id 5aa77a95ed36b51082d86d2e2b3d02dc448ede1d9f4ddc3f05f7ad774b2eaf43 Nov 04 01:10:31.025464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:31.025420 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" event={"ID":"a7f1bc94-14c3-4783-9aad-2efc85a264e1","Type":"ContainerStarted","Data":"5aa77a95ed36b51082d86d2e2b3d02dc448ede1d9f4ddc3f05f7ad774b2eaf43"} Nov 04 01:10:32.031476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.031438 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" event={"ID":"a7f1bc94-14c3-4783-9aad-2efc85a264e1","Type":"ContainerStarted","Data":"764e8e7e7b1c443e9da000df1d9500821318a2d28bf63d27de49d09eea84f9a0"} Nov 04 01:10:32.052049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.052003 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" podStartSLOduration=1.743535659 podStartE2EDuration="2.051988579s" totalImagesPullingTime="308.45292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:32.051200357 +0000 UTC m=+464.074132383" watchObservedRunningTime="2026-11-04 01:10:32.051988579 +0000 UTC m=+464.074920621" Nov 04 01:10:32.557679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.557629 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl"] Nov 04 01:10:32.561546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.561521 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.570970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.570891 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 01:10:32.572967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.572945 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl"] Nov 04 01:10:32.595616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.595580 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:32.629733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.629711 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.629851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.629737 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.629851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.629757 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.629947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.629850 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhkm\" (UniqueName: \"kubernetes.io/projected/870c4be8-527c-43ad-a63f-39dd02718c55-kube-api-access-fzhkm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.629947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.629903 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870c4be8-527c-43ad-a63f-39dd02718c55-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.629947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.629941 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731055 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731092 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731119 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731171 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhkm\" (UniqueName: \"kubernetes.io/projected/870c4be8-527c-43ad-a63f-39dd02718c55-kube-api-access-fzhkm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731202 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870c4be8-527c-43ad-a63f-39dd02718c55-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731257 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731525 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731581 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.731696 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.731670 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.733533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.733507 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870c4be8-527c-43ad-a63f-39dd02718c55-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.733699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.733676 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870c4be8-527c-43ad-a63f-39dd02718c55-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.743132 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.743111 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhkm\" (UniqueName: \"kubernetes.io/projected/870c4be8-527c-43ad-a63f-39dd02718c55-kube-api-access-fzhkm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k42tl\" (UID: \"870c4be8-527c-43ad-a63f-39dd02718c55\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:32.877891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:32.877825 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:33.150151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.150123 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl"] Nov 04 01:10:33.152596 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:10:33.152572 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870c4be8_527c_43ad_a63f_39dd02718c55.slice/crio-d4ab8aee46fec82cd4606d3bcffdd44d70979e8e52e3181b3792d1186ca31eb6 WatchSource:0}: Error finding container d4ab8aee46fec82cd4606d3bcffdd44d70979e8e52e3181b3792d1186ca31eb6: Status 404 returned error can't find the container with id d4ab8aee46fec82cd4606d3bcffdd44d70979e8e52e3181b3792d1186ca31eb6 Nov 04 01:10:33.670052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.670013 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26"] Nov 04 01:10:33.673612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.673547 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.681639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.681620 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 04 01:10:33.686773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.686731 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26"] Nov 04 01:10:33.740852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.740745 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.740852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.740783 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.740852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.740815 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mstr6\" (UniqueName: \"kubernetes.io/projected/61ca9501-204d-4daf-a56b-0794b0f8f6d0-kube-api-access-mstr6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.741082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.740975 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.741082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.741008 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.741170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.741129 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ca9501-204d-4daf-a56b-0794b0f8f6d0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842040 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842082 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842314 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842200 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mstr6\" (UniqueName: \"kubernetes.io/projected/61ca9501-204d-4daf-a56b-0794b0f8f6d0-kube-api-access-mstr6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842314 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842255 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842314 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842277 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842364 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ca9501-204d-4daf-a56b-0794b0f8f6d0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842476 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842545 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.842620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.842573 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.844640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.844610 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ca9501-204d-4daf-a56b-0794b0f8f6d0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.845169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.845146 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ca9501-204d-4daf-a56b-0794b0f8f6d0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.854790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.854771 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mstr6\" (UniqueName: \"kubernetes.io/projected/61ca9501-204d-4daf-a56b-0794b0f8f6d0-kube-api-access-mstr6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ghv26\" (UID: \"61ca9501-204d-4daf-a56b-0794b0f8f6d0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:33.991242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:33.991155 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:34.045121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:34.044454 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" event={"ID":"870c4be8-527c-43ad-a63f-39dd02718c55","Type":"ContainerStarted","Data":"cc01927a0563e0e2770f9c9d8aa758c81aeb5c9b0fc6cc7eb07719de951c01be"} Nov 04 01:10:34.045121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:34.044509 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" event={"ID":"870c4be8-527c-43ad-a63f-39dd02718c55","Type":"ContainerStarted","Data":"d4ab8aee46fec82cd4606d3bcffdd44d70979e8e52e3181b3792d1186ca31eb6"} Nov 04 01:10:34.067977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:34.067925 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" podStartSLOduration=1.8001377010000001 podStartE2EDuration="2.067905814s" totalImagesPullingTime="267.768113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:34.065154904 +0000 UTC m=+466.088086935" watchObservedRunningTime="2026-11-04 01:10:34.067905814 +0000 UTC m=+466.090837822" Nov 04 01:10:34.267398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:34.267370 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26"] Nov 04 01:10:34.269803 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:10:34.269774 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ca9501_204d_4daf_a56b_0794b0f8f6d0.slice/crio-7ad258918e1f0f81f97edc5a50b7fac65b76c037b4b07ca70673329a2aaf6b54 WatchSource:0}: Error finding container 7ad258918e1f0f81f97edc5a50b7fac65b76c037b4b07ca70673329a2aaf6b54: Status 404 returned error can't find the container with id 7ad258918e1f0f81f97edc5a50b7fac65b76c037b4b07ca70673329a2aaf6b54 Nov 04 01:10:34.878309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:34.878219 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:35.050150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:35.050108 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" event={"ID":"61ca9501-204d-4daf-a56b-0794b0f8f6d0","Type":"ContainerStarted","Data":"7313c43583a642ea8474dc3ac1f244f853ce4f7cec23a47c52699118cece2e76"} Nov 04 01:10:35.050319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:35.050176 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" event={"ID":"61ca9501-204d-4daf-a56b-0794b0f8f6d0","Type":"ContainerStarted","Data":"7ad258918e1f0f81f97edc5a50b7fac65b76c037b4b07ca70673329a2aaf6b54"} Nov 04 01:10:35.070973 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:35.070917 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" podStartSLOduration=1.777739038 podStartE2EDuration="2.070903064s" totalImagesPullingTime="293.164026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:35.069954485 +0000 UTC m=+467.092886494" watchObservedRunningTime="2026-11-04 01:10:35.070903064 +0000 UTC m=+467.093835070" Nov 04 01:10:35.991623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:35.991576 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:36.660358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.640146 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk"] Nov 04 01:10:36.678771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.678731 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk"] Nov 04 01:10:36.679098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.679079 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.684625 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.684599 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 04 01:10:36.780494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.780454 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.780641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.780509 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.780641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.780575 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zhf\" (UniqueName: \"kubernetes.io/projected/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-kube-api-access-22zhf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.780719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.780636 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.780719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.780665 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.780826 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.780807 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.881690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.881649 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.881885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.881710 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.881885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.881783 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.881885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.881832 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zhf\" (UniqueName: \"kubernetes.io/projected/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-kube-api-access-22zhf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.881885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.881863 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.882094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.881885 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.882230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.882208 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.882309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.882290 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.882515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.882489 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.884624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.884602 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.884933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.884917 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:36.894615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:36.894586 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zhf\" (UniqueName: \"kubernetes.io/projected/ba71124b-dc83-408b-8e5c-1b9abd6fac9a-kube-api-access-22zhf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk\" (UID: \"ba71124b-dc83-408b-8e5c-1b9abd6fac9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:37.002753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:37.002637 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:37.217742 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:37.217700 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:37.230068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:37.230036 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:37.284898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:37.284756 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk"] Nov 04 01:10:37.287094 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:10:37.287069 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba71124b_dc83_408b_8e5c_1b9abd6fac9a.slice/crio-dbdadd1bd9827b74e2f6d1ba8d9e5e38d9c298c24e7f061d6645e86f1c56ab96 WatchSource:0}: Error finding container dbdadd1bd9827b74e2f6d1ba8d9e5e38d9c298c24e7f061d6645e86f1c56ab96: Status 404 returned error can't find the container with id dbdadd1bd9827b74e2f6d1ba8d9e5e38d9c298c24e7f061d6645e86f1c56ab96 Nov 04 01:10:38.068614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:38.068578 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" event={"ID":"ba71124b-dc83-408b-8e5c-1b9abd6fac9a","Type":"ContainerStarted","Data":"398704894b70c907f45fec22c3bb6c9fe091c1d63192e1030ea9ef8f588623a8"} Nov 04 01:10:38.069037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:38.068628 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" event={"ID":"ba71124b-dc83-408b-8e5c-1b9abd6fac9a","Type":"ContainerStarted","Data":"dbdadd1bd9827b74e2f6d1ba8d9e5e38d9c298c24e7f061d6645e86f1c56ab96"} Nov 04 01:10:38.079963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:38.079939 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:10:38.091799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:38.091756 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" podStartSLOduration=1.807016331 podStartE2EDuration="2.091742895s" totalImagesPullingTime="284.726564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:38.090583188 +0000 UTC m=+470.113515213" watchObservedRunningTime="2026-11-04 01:10:38.091742895 +0000 UTC m=+470.114674901" Nov 04 01:10:38.197647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:38.197612 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:38.209877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:38.209855 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:39.003763 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:39.003715 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:39.093098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:39.093044 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:10:40.595227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:40.595195 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:40.607877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:40.607854 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:41.097440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:41.097409 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:10:42.878964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:42.878932 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:42.891511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:42.891486 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:43.106754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:43.106729 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" Nov 04 01:10:43.991774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:43.991744 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:44.004571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:44.004542 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:44.111064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:44.111028 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" Nov 04 01:10:47.002827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:47.002755 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:47.015376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:47.015348 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:47.123950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:10:47.123922 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" Nov 04 01:10:59.456756 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:10:59.456667 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:11:15.678698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.678662 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf"] Nov 04 01:11:15.684113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.684090 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.692927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.692905 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4zhmf\"" Nov 04 01:11:15.695224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.695202 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf"] Nov 04 01:11:15.812718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.812675 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.812918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.812730 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.812918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.812812 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee19db5-0a09-4b15-b268-e749a0a2969f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.812918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.812856 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkwl\" (UniqueName: \"kubernetes.io/projected/2ee19db5-0a09-4b15-b268-e749a0a2969f-kube-api-access-wgkwl\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.812918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.812890 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.813115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.812965 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.813115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.812994 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.813115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.813014 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.813115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.813032 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914126 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914172 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914203 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914222 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914239 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914277 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914310 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914367 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee19db5-0a09-4b15-b268-e749a0a2969f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914393 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkwl\" (UniqueName: \"kubernetes.io/projected/2ee19db5-0a09-4b15-b268-e749a0a2969f-kube-api-access-wgkwl\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914675 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914767 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.914864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914807 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.915000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.914976 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.915200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.915182 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee19db5-0a09-4b15-b268-e749a0a2969f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.916515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.916494 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.916744 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.916719 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.926993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.926967 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee19db5-0a09-4b15-b268-e749a0a2969f-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.927075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.926980 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkwl\" (UniqueName: \"kubernetes.io/projected/2ee19db5-0a09-4b15-b268-e749a0a2969f-kube-api-access-wgkwl\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-kvnxf\" (UID: \"2ee19db5-0a09-4b15-b268-e749a0a2969f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:15.998175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:15.998113 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:16.277460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:16.277435 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf"] Nov 04 01:11:16.279707 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:11:16.279680 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee19db5_0a09_4b15_b268_e749a0a2969f.slice/crio-eecce9fc373a4c0f96d9b9510bfdd6ea9f2b24f585621a7b1480e45cfb2586fc WatchSource:0}: Error finding container eecce9fc373a4c0f96d9b9510bfdd6ea9f2b24f585621a7b1480e45cfb2586fc: Status 404 returned error can't find the container with id eecce9fc373a4c0f96d9b9510bfdd6ea9f2b24f585621a7b1480e45cfb2586fc Nov 04 01:11:17.241419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:17.241379 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" event={"ID":"2ee19db5-0a09-4b15-b268-e749a0a2969f","Type":"ContainerStarted","Data":"eecce9fc373a4c0f96d9b9510bfdd6ea9f2b24f585621a7b1480e45cfb2586fc"} Nov 04 01:11:19.823782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:19.823749 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:19.824080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:19.823818 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:19.824080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:19.823849 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:20.280814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:20.280778 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" event={"ID":"2ee19db5-0a09-4b15-b268-e749a0a2969f","Type":"ContainerStarted","Data":"14710f91c139622a4b96991028928b30c89b467d15ee6126fc1ece08c14ee089"} Nov 04 01:11:20.305564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:20.305517 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podStartSLOduration=1.764270413 podStartE2EDuration="5.305503078s" totalImagesPullingTime="3.541232665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:20.304093984 +0000 UTC m=+512.327026012" watchObservedRunningTime="2026-11-04 01:11:20.305503078 +0000 UTC m=+512.328435085" Nov 04 01:11:20.998957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:20.998914 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:21.000532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:21.000506 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:11:21.000659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:21.000557 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 01:11:21.606317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:21.606279 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b5bdbf74f-lkjbp"] Nov 04 01:11:21.606569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:21.606540 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" podUID="1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" containerName="authorino" containerID="cri-o://5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3" gracePeriod=30 Nov 04 01:11:21.898306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:21.898277 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:11:21.999302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:21.999270 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:11:21.999822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:21.999321 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 01:11:22.087612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.087579 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-tls-cert\" (UniqueName: \"kubernetes.io/secret/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-tls-cert\") pod \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " Nov 04 01:11:22.087790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.087629 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-kube-api-access-5vhvp\" (UniqueName: \"kubernetes.io/projected/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-kube-api-access-5vhvp\") pod \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\" (UID: \"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc\") " Nov 04 01:11:22.089854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.089829 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-kube-api-access-5vhvp" pod "1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" (UID: "1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc"). InnerVolumeSpecName "kube-api-access-5vhvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:11:22.098199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.098158 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-tls-cert" pod "1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" (UID: "1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:11:22.184547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.184471 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm"] Nov 04 01:11:22.184863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.184850 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" containerName="authorino" Nov 04 01:11:22.184900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.184863 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" containerName="authorino" Nov 04 01:11:22.184950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.184941 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" containerName="authorino" Nov 04 01:11:22.188174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.188149 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.188521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.188494 4312 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-tls-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:22.188670 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.188527 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vhvp\" (UniqueName: \"kubernetes.io/projected/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc-kube-api-access-5vhvp\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:22.194719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.194696 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-b2t89\"" Nov 04 01:11:22.194881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.194859 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 01:11:22.195324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.194706 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 01:11:22.197547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.197523 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm"] Nov 04 01:11:22.289850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.289816 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqlg\" (UniqueName: \"kubernetes.io/projected/b1c8f311-98e8-4921-82ab-8426adcdd03c-kube-api-access-8bqlg\") pod \"ai-gateway-controller-7cb8f6bb64-q67lm\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.290037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.289861 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c8f311-98e8-4921-82ab-8426adcdd03c-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-q67lm\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.300666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.300630 4312 generic.go:356] "Generic (PLEG): container finished" podID="1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" containerID="5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3" exitCode=0 Nov 04 01:11:22.300822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.300685 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" event={"ID":"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc","Type":"ContainerDied","Data":"5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3"} Nov 04 01:11:22.300822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.300724 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" event={"ID":"1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc","Type":"ContainerDied","Data":"401fbe59306ddbca552c87a2088afea85a377ca7ff918bf9ec8ea63cdb833205"} Nov 04 01:11:22.300822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.300740 4312 scope.go:122] "RemoveContainer" containerID="5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3" Nov 04 01:11:22.300822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.300695 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b5bdbf74f-lkjbp" Nov 04 01:11:22.309419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.309403 4312 scope.go:122] "RemoveContainer" containerID="5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3" Nov 04 01:11:22.309687 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:11:22.309664 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3\": container with ID starting with 5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3 not found: ID does not exist" containerID="5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3" Nov 04 01:11:22.309793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.309691 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3"} err="failed to get container status \"5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3\": rpc error: code = NotFound desc = could not find container \"5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3\": container with ID starting with 5c4458aa77bd032c3fcbee7bdd6c6bf9fa28f759198d3768a1716626e651b4b3 not found: ID does not exist" Nov 04 01:11:22.318199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.318177 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b5bdbf74f-lkjbp"] Nov 04 01:11:22.322106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.322080 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b5bdbf74f-lkjbp"] Nov 04 01:11:22.391015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.390981 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqlg\" (UniqueName: \"kubernetes.io/projected/b1c8f311-98e8-4921-82ab-8426adcdd03c-kube-api-access-8bqlg\") pod \"ai-gateway-controller-7cb8f6bb64-q67lm\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.391187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.391022 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c8f311-98e8-4921-82ab-8426adcdd03c-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-q67lm\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.393544 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.393519 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c8f311-98e8-4921-82ab-8426adcdd03c-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-q67lm\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.404012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.403989 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqlg\" (UniqueName: \"kubernetes.io/projected/b1c8f311-98e8-4921-82ab-8426adcdd03c-kube-api-access-8bqlg\") pod \"ai-gateway-controller-7cb8f6bb64-q67lm\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.501411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.501320 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:22.801503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.801476 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm"] Nov 04 01:11:22.803619 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:11:22.803588 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c8f311_98e8_4921_82ab_8426adcdd03c.slice/crio-2c9116e2216a219a41dd765f306b2848d730b373dbbeea7dfddcf8ba44411aec WatchSource:0}: Error finding container 2c9116e2216a219a41dd765f306b2848d730b373dbbeea7dfddcf8ba44411aec: Status 404 returned error can't find the container with id 2c9116e2216a219a41dd765f306b2848d730b373dbbeea7dfddcf8ba44411aec Nov 04 01:11:22.998729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.998693 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:11:22.998910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:22.998744 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 01:11:23.306977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:23.306938 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" event={"ID":"b1c8f311-98e8-4921-82ab-8426adcdd03c","Type":"ContainerStarted","Data":"2c9116e2216a219a41dd765f306b2848d730b373dbbeea7dfddcf8ba44411aec"} Nov 04 01:11:23.998910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:23.998879 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:11:23.999084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:23.998937 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 01:11:24.106576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:24.106542 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc" path="/var/lib/kubelet/pods/1ed660ad-f2c7-4a1c-adae-773cd5f0a6cc/volumes" Nov 04 01:11:24.999442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:24.999400 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:11:24.999854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:24.999458 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 01:11:25.999196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:25.999159 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:11:25.999396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:25.999214 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 01:11:26.999110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:26.999069 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:11:26.999565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:26.999139 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 01:11:28.293195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.293156 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn"] Nov 04 01:11:28.297365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.297343 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.303884 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.303858 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 01:11:28.304988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.303958 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gczb8\"" Nov 04 01:11:28.308625 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.308552 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn"] Nov 04 01:11:28.325058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.325033 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-lggqw"] Nov 04 01:11:28.329033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.329012 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.332907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.332885 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" event={"ID":"b1c8f311-98e8-4921-82ab-8426adcdd03c","Type":"ContainerStarted","Data":"9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177"} Nov 04 01:11:28.333157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.333139 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:28.340606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.340583 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-lggqw"] Nov 04 01:11:28.341177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.341151 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfjt\" (UniqueName: \"kubernetes.io/projected/5d475784-6dfc-4d23-8e67-70c7cca43636-kube-api-access-hgfjt\") pod \"payload-processing-7bb9489676-lggqw\" (UID: \"5d475784-6dfc-4d23-8e67-70c7cca43636\") " pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.341298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.341274 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmpm5\" (UniqueName: \"kubernetes.io/projected/a0d81c57-ee60-4be3-b9b2-e0d10514088c-kube-api-access-kmpm5\") pod \"payload-pre-processing-7b98bb678c-4xmvn\" (UID: \"a0d81c57-ee60-4be3-b9b2-e0d10514088c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.341382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.341308 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d475784-6dfc-4d23-8e67-70c7cca43636-plugins-config-volume\") pod \"payload-processing-7bb9489676-lggqw\" (UID: \"5d475784-6dfc-4d23-8e67-70c7cca43636\") " pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.341456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.341414 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d81c57-ee60-4be3-b9b2-e0d10514088c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-4xmvn\" (UID: \"a0d81c57-ee60-4be3-b9b2-e0d10514088c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.369849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.369792 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" podStartSLOduration=1.854849714 podStartE2EDuration="6.369773987s" totalImagesPullingTime="4.514924273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:28.368707195 +0000 UTC m=+520.391639203" watchObservedRunningTime="2026-11-04 01:11:28.369773987 +0000 UTC m=+520.392705994" Nov 04 01:11:28.442954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.442907 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfjt\" (UniqueName: \"kubernetes.io/projected/5d475784-6dfc-4d23-8e67-70c7cca43636-kube-api-access-hgfjt\") pod \"payload-processing-7bb9489676-lggqw\" (UID: \"5d475784-6dfc-4d23-8e67-70c7cca43636\") " pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.443147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.443007 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmpm5\" (UniqueName: \"kubernetes.io/projected/a0d81c57-ee60-4be3-b9b2-e0d10514088c-kube-api-access-kmpm5\") pod \"payload-pre-processing-7b98bb678c-4xmvn\" (UID: \"a0d81c57-ee60-4be3-b9b2-e0d10514088c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.443147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.443033 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d475784-6dfc-4d23-8e67-70c7cca43636-plugins-config-volume\") pod \"payload-processing-7bb9489676-lggqw\" (UID: \"5d475784-6dfc-4d23-8e67-70c7cca43636\") " pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.443147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.443078 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d81c57-ee60-4be3-b9b2-e0d10514088c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-4xmvn\" (UID: \"a0d81c57-ee60-4be3-b9b2-e0d10514088c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.443881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.443854 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d475784-6dfc-4d23-8e67-70c7cca43636-plugins-config-volume\") pod \"payload-processing-7bb9489676-lggqw\" (UID: \"5d475784-6dfc-4d23-8e67-70c7cca43636\") " pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.444060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.444032 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d81c57-ee60-4be3-b9b2-e0d10514088c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-4xmvn\" (UID: \"a0d81c57-ee60-4be3-b9b2-e0d10514088c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.455264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.455239 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfjt\" (UniqueName: \"kubernetes.io/projected/5d475784-6dfc-4d23-8e67-70c7cca43636-kube-api-access-hgfjt\") pod \"payload-processing-7bb9489676-lggqw\" (UID: \"5d475784-6dfc-4d23-8e67-70c7cca43636\") " pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.455463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.455441 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmpm5\" (UniqueName: \"kubernetes.io/projected/a0d81c57-ee60-4be3-b9b2-e0d10514088c-kube-api-access-kmpm5\") pod \"payload-pre-processing-7b98bb678c-4xmvn\" (UID: \"a0d81c57-ee60-4be3-b9b2-e0d10514088c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.614524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.614450 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:28.645343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.645293 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:28.907957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.907929 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn"] Nov 04 01:11:28.910518 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:11:28.910486 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d81c57_ee60_4be3_b9b2_e0d10514088c.slice/crio-9dc15d699c1c9f6c1b54161ba5f8883c5a322c1bcb74a101aa6eeb88ab14aef7 WatchSource:0}: Error finding container 9dc15d699c1c9f6c1b54161ba5f8883c5a322c1bcb74a101aa6eeb88ab14aef7: Status 404 returned error can't find the container with id 9dc15d699c1c9f6c1b54161ba5f8883c5a322c1bcb74a101aa6eeb88ab14aef7 Nov 04 01:11:28.937385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.937358 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-lggqw"] Nov 04 01:11:28.939068 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:11:28.939017 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d475784_6dfc_4d23_8e67_70c7cca43636.slice/crio-ff231418b391a9054ea84780b8f057ebeae62edb8fd7742ff9389c8563909c4a WatchSource:0}: Error finding container ff231418b391a9054ea84780b8f057ebeae62edb8fd7742ff9389c8563909c4a: Status 404 returned error can't find the container with id ff231418b391a9054ea84780b8f057ebeae62edb8fd7742ff9389c8563909c4a Nov 04 01:11:28.999199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.999174 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:11:28.999319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:28.999244 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:11:29.338900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:29.338860 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-lggqw" event={"ID":"5d475784-6dfc-4d23-8e67-70c7cca43636","Type":"ContainerStarted","Data":"ff231418b391a9054ea84780b8f057ebeae62edb8fd7742ff9389c8563909c4a"} Nov 04 01:11:29.340090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:29.340067 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" event={"ID":"a0d81c57-ee60-4be3-b9b2-e0d10514088c","Type":"ContainerStarted","Data":"9dc15d699c1c9f6c1b54161ba5f8883c5a322c1bcb74a101aa6eeb88ab14aef7"} Nov 04 01:11:29.456371 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:11:29.456325 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:11:30.041554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.041512 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:11:30.041771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.041586 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:11:30.719411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.717404 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk"] Nov 04 01:11:30.721577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.721554 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.740129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.738967 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk"] Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775638 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775691 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775732 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775771 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775798 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775868 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775907 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvxx\" (UniqueName: \"kubernetes.io/projected/33e3cbdc-f101-49ff-953e-c46a13d2658e-kube-api-access-sgvxx\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.776185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.775946 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e3cbdc-f101-49ff-953e-c46a13d2658e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.876680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876644 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.876871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876702 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.876871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876744 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.876871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876770 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.876871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876807 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.876871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876842 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.877156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876879 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvxx\" (UniqueName: \"kubernetes.io/projected/33e3cbdc-f101-49ff-953e-c46a13d2658e-kube-api-access-sgvxx\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.877156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876905 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e3cbdc-f101-49ff-953e-c46a13d2658e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.877156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.876964 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.877771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.877364 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.877980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.877955 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.878062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.878036 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.878161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.878136 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e3cbdc-f101-49ff-953e-c46a13d2658e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.878327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.878296 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.881792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.881763 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.882804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.882778 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.895258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.895223 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvxx\" (UniqueName: \"kubernetes.io/projected/33e3cbdc-f101-49ff-953e-c46a13d2658e-kube-api-access-sgvxx\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.896748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.896722 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e3cbdc-f101-49ff-953e-c46a13d2658e-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-4l6sk\" (UID: \"33e3cbdc-f101-49ff-953e-c46a13d2658e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:30.999511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:30.998652 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:31.041556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.041519 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:31.086596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.085898 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:11:31.086596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.085951 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:11:31.376627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.376574 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk"] Nov 04 01:11:31.382236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.382198 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:31.382373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.382287 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:31.382373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.382349 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:31.469105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.468891 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:31.472045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:31.471942 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" Nov 04 01:11:32.381574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:32.381528 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" event={"ID":"a0d81c57-ee60-4be3-b9b2-e0d10514088c","Type":"ContainerStarted","Data":"90f73eddf3ff1b5dda97fe6aaf5e4df3619dcb91674454b100db99eb5cd3b4cc"} Nov 04 01:11:32.381985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:32.381613 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:32.383376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:32.383324 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" event={"ID":"33e3cbdc-f101-49ff-953e-c46a13d2658e","Type":"ContainerStarted","Data":"e1ecaa6bd3dc9047ff4db2f4c9dc60df80f41b2e55c15b8f47fdf77f9904e8a5"} Nov 04 01:11:32.383498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:32.383390 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" event={"ID":"33e3cbdc-f101-49ff-953e-c46a13d2658e","Type":"ContainerStarted","Data":"2fe1ed2f255494f9e4e5782a0635921f38b3a0b1030ff24aa4b780b55f7d4382"} Nov 04 01:11:32.403916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:32.403873 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" podStartSLOduration=1.085540083 podStartE2EDuration="4.403861825s" totalImagesPullingTime="3.318321742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:32.401529349 +0000 UTC m=+524.424461355" watchObservedRunningTime="2026-11-04 01:11:32.403861825 +0000 UTC m=+524.426793832" Nov 04 01:11:32.424602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:32.424564 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podStartSLOduration=2.424552425 podStartE2EDuration="2.424552425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:32.423489955 +0000 UTC m=+524.446421976" watchObservedRunningTime="2026-11-04 01:11:32.424552425 +0000 UTC m=+524.447484432" Nov 04 01:11:33.041833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:33.041796 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:33.042911 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:33.042890 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:33.042959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:33.042936 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 01:11:33.389014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:33.388927 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-lggqw" event={"ID":"5d475784-6dfc-4d23-8e67-70c7cca43636","Type":"ContainerStarted","Data":"613eb7ff3534763f2fca8e6658d1eb94287545fdde501bd5a24cf1b9ca341600"} Nov 04 01:11:33.389473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:33.389312 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:33.408115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:33.408069 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-lggqw" podStartSLOduration=2.028710501 podStartE2EDuration="5.408054841s" totalImagesPullingTime="3.37934434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:33.407296654 +0000 UTC m=+525.430228662" watchObservedRunningTime="2026-11-04 01:11:33.408054841 +0000 UTC m=+525.430986871" Nov 04 01:11:34.042410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:34.042373 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:34.042581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:34.042441 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 01:11:35.042027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:35.041989 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:35.042416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:35.042046 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 01:11:36.042477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:36.042445 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:36.042831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:36.042495 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 01:11:37.042713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:37.042680 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:37.043061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:37.042731 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 01:11:38.042043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:38.042008 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:38.042197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:38.042059 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 01:11:39.042457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.042424 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:39.042867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.042477 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 01:11:39.342681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.342604 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:39.628924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.628848 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-568986c44-m79cq"] Nov 04 01:11:39.632759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.632739 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:39.643657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.643633 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-568986c44-m79cq"] Nov 04 01:11:39.664961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.664937 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/64ec9c7b-6735-43a7-a859-e7186474c121-kube-api-access-2wdwr\") pod \"ai-gateway-controller-568986c44-m79cq\" (UID: \"64ec9c7b-6735-43a7-a859-e7186474c121\") " pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:39.665078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.665006 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ec9c7b-6735-43a7-a859-e7186474c121-metrics-certs\") pod \"ai-gateway-controller-568986c44-m79cq\" (UID: \"64ec9c7b-6735-43a7-a859-e7186474c121\") " pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:39.766224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.766190 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/64ec9c7b-6735-43a7-a859-e7186474c121-kube-api-access-2wdwr\") pod \"ai-gateway-controller-568986c44-m79cq\" (UID: \"64ec9c7b-6735-43a7-a859-e7186474c121\") " pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:39.766392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.766277 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ec9c7b-6735-43a7-a859-e7186474c121-metrics-certs\") pod \"ai-gateway-controller-568986c44-m79cq\" (UID: \"64ec9c7b-6735-43a7-a859-e7186474c121\") " pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:39.768847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.768816 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ec9c7b-6735-43a7-a859-e7186474c121-metrics-certs\") pod \"ai-gateway-controller-568986c44-m79cq\" (UID: \"64ec9c7b-6735-43a7-a859-e7186474c121\") " pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:39.778584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.778559 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/64ec9c7b-6735-43a7-a859-e7186474c121-kube-api-access-2wdwr\") pod \"ai-gateway-controller-568986c44-m79cq\" (UID: \"64ec9c7b-6735-43a7-a859-e7186474c121\") " pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:39.947621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:39.947536 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:40.232434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:40.232391 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-568986c44-m79cq"] Nov 04 01:11:40.235785 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:11:40.235751 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ec9c7b_6735_43a7_a859_e7186474c121.slice/crio-2d6a3e4b4d1a02908eef0cd6ec1c111cc661f4e749aa77aab0878ebe33515163 WatchSource:0}: Error finding container 2d6a3e4b4d1a02908eef0cd6ec1c111cc661f4e749aa77aab0878ebe33515163: Status 404 returned error can't find the container with id 2d6a3e4b4d1a02908eef0cd6ec1c111cc661f4e749aa77aab0878ebe33515163 Nov 04 01:11:40.343120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:40.343064 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:40.343493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:40.343466 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:40.344237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:40.344222 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" Nov 04 01:11:40.424416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:40.424379 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-568986c44-m79cq" event={"ID":"64ec9c7b-6735-43a7-a859-e7186474c121","Type":"ContainerStarted","Data":"2d6a3e4b4d1a02908eef0cd6ec1c111cc661f4e749aa77aab0878ebe33515163"} Nov 04 01:11:41.431443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:41.431410 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-568986c44-m79cq" event={"ID":"64ec9c7b-6735-43a7-a859-e7186474c121","Type":"ContainerStarted","Data":"4245c3cf4bdc89dcb48ea6154b7c11554020554c47b661c035b2fc037928ca23"} Nov 04 01:11:41.431830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:41.431490 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:41.452182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:41.452134 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-568986c44-m79cq" podStartSLOduration=1.985608226 podStartE2EDuration="2.452121907s" totalImagesPullingTime="466.513681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:41.450052442 +0000 UTC m=+533.472984449" watchObservedRunningTime="2026-11-04 01:11:41.452121907 +0000 UTC m=+533.475053913" Nov 04 01:11:43.393065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:43.393033 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4xmvn" Nov 04 01:11:45.401724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.401695 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-lggqw" Nov 04 01:11:45.733790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.730798 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5"] Nov 04 01:11:45.736905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.736877 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.749355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.749304 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5"] Nov 04 01:11:45.824605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824573 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.824734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824613 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.824734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824637 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a085702c-fd6e-46d0-b282-1f96cf558e24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.824812 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824735 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.824812 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824766 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.824920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824897 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.824955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824927 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.824987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.824953 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.825035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.825023 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnf4g\" (UniqueName: \"kubernetes.io/projected/a085702c-fd6e-46d0-b282-1f96cf558e24-kube-api-access-gnf4g\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.925967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.925936 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.925971 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.925995 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926016 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnf4g\" (UniqueName: \"kubernetes.io/projected/a085702c-fd6e-46d0-b282-1f96cf558e24-kube-api-access-gnf4g\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926067 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926083 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926105 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a085702c-fd6e-46d0-b282-1f96cf558e24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926134 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926160 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926418 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926444 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926512 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926626 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.926880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.926855 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a085702c-fd6e-46d0-b282-1f96cf558e24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.928433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.928403 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.928702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.928684 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.938328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.938310 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a085702c-fd6e-46d0-b282-1f96cf558e24-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:45.938486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:45.938468 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnf4g\" (UniqueName: \"kubernetes.io/projected/a085702c-fd6e-46d0-b282-1f96cf558e24-kube-api-access-gnf4g\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-ng8g5\" (UID: \"a085702c-fd6e-46d0-b282-1f96cf558e24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:46.051953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.051927 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:46.334589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.334564 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5"] Nov 04 01:11:46.336381 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:11:46.336350 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda085702c_fd6e_46d0_b282_1f96cf558e24.slice/crio-f8f51dd497fb4e644f33f8dfac919f5577c57f03c064aca2a0f662da099f6f62 WatchSource:0}: Error finding container f8f51dd497fb4e644f33f8dfac919f5577c57f03c064aca2a0f662da099f6f62: Status 404 returned error can't find the container with id f8f51dd497fb4e644f33f8dfac919f5577c57f03c064aca2a0f662da099f6f62 Nov 04 01:11:46.339475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.339446 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:46.339545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.339512 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:46.339591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.339541 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:11:46.458856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.458822 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" event={"ID":"a085702c-fd6e-46d0-b282-1f96cf558e24","Type":"ContainerStarted","Data":"d8ac115403f6d3cd0c93e905feb58350f0eb96a2423ac375b9f00397b1f3f619"} Nov 04 01:11:46.459237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.458886 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" event={"ID":"a085702c-fd6e-46d0-b282-1f96cf558e24","Type":"ContainerStarted","Data":"f8f51dd497fb4e644f33f8dfac919f5577c57f03c064aca2a0f662da099f6f62"} Nov 04 01:11:46.484639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:46.484581 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podStartSLOduration=1.4845620990000001 podStartE2EDuration="1.484562099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:46.48159231 +0000 UTC m=+538.504524329" watchObservedRunningTime="2026-11-04 01:11:46.484562099 +0000 UTC m=+538.507494112" Nov 04 01:11:47.064987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:47.059706 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:47.069801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:47.069565 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:47.069801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:47.069625 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 01:11:48.066358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:48.059762 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:48.066358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:48.059824 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 01:11:49.072357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:49.071441 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:49.072357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:49.071505 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 01:11:50.062357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:50.056710 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:50.062357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:50.056771 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 01:11:51.072462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:51.072424 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:51.072921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:51.072485 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 01:11:52.064797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.064757 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:52.065086 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.065054 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 01:11:52.439569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.439481 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-568986c44-m79cq" Nov 04 01:11:52.492553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.492512 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm"] Nov 04 01:11:52.492858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.492780 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" podUID="b1c8f311-98e8-4921-82ab-8426adcdd03c" containerName="manager" containerID="cri-o://9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177" gracePeriod=30 Nov 04 01:11:52.753226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.753192 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:52.919725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.919689 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c8f311-98e8-4921-82ab-8426adcdd03c-kube-api-access-8bqlg\" (UniqueName: \"kubernetes.io/projected/b1c8f311-98e8-4921-82ab-8426adcdd03c-kube-api-access-8bqlg\") pod \"b1c8f311-98e8-4921-82ab-8426adcdd03c\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " Nov 04 01:11:52.919919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.919734 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c8f311-98e8-4921-82ab-8426adcdd03c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c8f311-98e8-4921-82ab-8426adcdd03c-metrics-certs\") pod \"b1c8f311-98e8-4921-82ab-8426adcdd03c\" (UID: \"b1c8f311-98e8-4921-82ab-8426adcdd03c\") " Nov 04 01:11:52.921967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.921943 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c8f311-98e8-4921-82ab-8426adcdd03c-metrics-certs" pod "b1c8f311-98e8-4921-82ab-8426adcdd03c" (UID: "b1c8f311-98e8-4921-82ab-8426adcdd03c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:11:52.922068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:52.921987 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c8f311-98e8-4921-82ab-8426adcdd03c-kube-api-access-8bqlg" pod "b1c8f311-98e8-4921-82ab-8426adcdd03c" (UID: "b1c8f311-98e8-4921-82ab-8426adcdd03c"). InnerVolumeSpecName "kube-api-access-8bqlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:11:53.021285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.021209 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bqlg\" (UniqueName: \"kubernetes.io/projected/b1c8f311-98e8-4921-82ab-8426adcdd03c-kube-api-access-8bqlg\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:53.021285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.021238 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c8f311-98e8-4921-82ab-8426adcdd03c-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:53.052772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.052735 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:53.052875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.052803 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 01:11:53.503690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.503656 4312 generic.go:356] "Generic (PLEG): container finished" podID="b1c8f311-98e8-4921-82ab-8426adcdd03c" containerID="9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177" exitCode=0 Nov 04 01:11:53.504182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.503701 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" event={"ID":"b1c8f311-98e8-4921-82ab-8426adcdd03c","Type":"ContainerDied","Data":"9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177"} Nov 04 01:11:53.504182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.503709 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" Nov 04 01:11:53.504182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.503753 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm" event={"ID":"b1c8f311-98e8-4921-82ab-8426adcdd03c","Type":"ContainerDied","Data":"2c9116e2216a219a41dd765f306b2848d730b373dbbeea7dfddcf8ba44411aec"} Nov 04 01:11:53.504182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.503777 4312 scope.go:122] "RemoveContainer" containerID="9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177" Nov 04 01:11:53.512934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.512915 4312 scope.go:122] "RemoveContainer" containerID="9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177" Nov 04 01:11:53.513252 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:11:53.513232 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177\": container with ID starting with 9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177 not found: ID does not exist" containerID="9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177" Nov 04 01:11:53.513309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.513259 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177"} err="failed to get container status \"9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177\": rpc error: code = NotFound desc = could not find container \"9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177\": container with ID starting with 9fd87251e68a6728b9a2180ac835c03a6d40a662060af494bf176954db24b177 not found: ID does not exist" Nov 04 01:11:53.526498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.526472 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm"] Nov 04 01:11:53.528365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:53.528348 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm"] Nov 04 01:11:54.111146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:54.111112 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c8f311-98e8-4921-82ab-8426adcdd03c" path="/var/lib/kubelet/pods/b1c8f311-98e8-4921-82ab-8426adcdd03c/volumes" Nov 04 01:11:54.852466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:54.852434 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:54.853047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:54.853018 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:54.853663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:11:54.853645 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" Nov 04 01:11:59.456842 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:11:59.456800 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:12:29.456371 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:12:29.456275 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:12:31.437074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.437043 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl"] Nov 04 01:12:31.437594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.437569 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b1c8f311-98e8-4921-82ab-8426adcdd03c" containerName="manager" Nov 04 01:12:31.437594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.437593 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c8f311-98e8-4921-82ab-8426adcdd03c" containerName="manager" Nov 04 01:12:31.437828 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.437691 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c8f311-98e8-4921-82ab-8426adcdd03c" containerName="manager" Nov 04 01:12:31.440968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.440948 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.447892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.447869 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a5115a-openshift-default-dockercfg-fx5s6\"" Nov 04 01:12:31.455227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.455204 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl"] Nov 04 01:12:31.599875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.599847 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7"] Nov 04 01:12:31.601798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.601763 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.601798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.601792 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3359d44-f1ed-463c-a61e-52f2ce59795d-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.601994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.601885 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.601994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.601978 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.602109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.602049 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.602109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.602097 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx59g\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-kube-api-access-dx59g\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.602263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.602148 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.602263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.602171 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.602263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.602192 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.603462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.603444 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.643727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.643705 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-9902c3-openshift-default-dockercfg-5blnq\"" Nov 04 01:12:31.649094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.649067 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7"] Nov 04 01:12:31.703907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.703827 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.703907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.703866 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.703907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.703884 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.703937 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2gs\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-kube-api-access-gd2gs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704086 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704128 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704155 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3359d44-f1ed-463c-a61e-52f2ce59795d-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704186 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704221 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704248 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704421 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704479 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704496 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704513 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704566 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704579 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704622 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704662 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704713 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx59g\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-kube-api-access-dx59g\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.704820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704748 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.704820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.704749 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.705033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.705011 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3359d44-f1ed-463c-a61e-52f2ce59795d-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.705092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.705013 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.706411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.706392 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.706516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.706445 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.718745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.718714 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.718848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.718788 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx59g\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-kube-api-access-dx59g\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.754410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.754378 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:31.805699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.805668 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.805699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.805703 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2gs\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-kube-api-access-gd2gs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.805909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.805818 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.805984 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806031 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806086 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806101 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806104 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806125 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806215 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806279 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806493 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806597 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.806712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.806695 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.808477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.808388 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.811756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.811409 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.822369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.822232 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.824549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.822756 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2gs\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-kube-api-access-gd2gs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.844890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.844792 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv"] Nov 04 01:12:31.849813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.849789 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.864674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.864489 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cffa80-openshift-default-dockercfg-s6kbq\"" Nov 04 01:12:31.867012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.866988 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv"] Nov 04 01:12:31.907394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907363 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-kube-api-access-r7pb9\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907455 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907526 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638bc5b7-a66e-43d9-8053-b754b0b00afc-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907630 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907689 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907673 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907691 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907763 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907751 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907773 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.907833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.907806 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:31.918187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.918160 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:31.955070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.954990 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt"] Nov 04 01:12:31.960350 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.960306 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:31.970809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.970210 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1036e8-openshift-default-dockercfg-wbg6m\"" Nov 04 01:12:31.975072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:31.974226 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt"] Nov 04 01:12:32.008384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008352 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.008549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008403 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-kube-api-access-r7pb9\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.008549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008445 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.008549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008470 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56c2\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-kube-api-access-x56c2\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.008549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008502 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.008549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008528 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.008799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008552 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638bc5b7-a66e-43d9-8053-b754b0b00afc-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.008799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008600 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f1428-b1b5-40cd-851f-f3de8c0056a4-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.008799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008640 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.008799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008701 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.008799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008739 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.008799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008785 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008811 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.009063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008818 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008851 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.008981 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.009155 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.009223 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.009279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.009236 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.009278 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.009370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.009313 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.009426 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.009545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.009525 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638bc5b7-a66e-43d9-8053-b754b0b00afc-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.011538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.011506 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.011683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.011649 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.027692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.027665 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-kube-api-access-r7pb9\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.028733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.028709 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.043042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.043010 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl"] Nov 04 01:12:32.045025 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:32.045000 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3359d44_f1ed_463c_a61e_52f2ce59795d.slice/crio-17db6f2485a924c882faac7e488008e5e4c51b2640cb1b0c55ca0c936b448072 WatchSource:0}: Error finding container 17db6f2485a924c882faac7e488008e5e4c51b2640cb1b0c55ca0c936b448072: Status 404 returned error can't find the container with id 17db6f2485a924c882faac7e488008e5e4c51b2640cb1b0c55ca0c936b448072 Nov 04 01:12:32.047686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.047657 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.047770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.047720 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.047770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.047747 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.110936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.110905 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.110947 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56c2\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-kube-api-access-x56c2\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.110986 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111033 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f1428-b1b5-40cd-851f-f3de8c0056a4-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111064 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111120 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111155 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111250 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111464 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111280 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.112087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.111631 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.113233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.112849 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.113414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.113250 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f1428-b1b5-40cd-851f-f3de8c0056a4-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.113694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.113676 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.114501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.114481 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.120654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.120630 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.135191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.135153 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.136437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.136420 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56c2\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-kube-api-access-x56c2\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.154391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.151368 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5"] Nov 04 01:12:32.155757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.155737 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.165424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.165396 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5"] Nov 04 01:12:32.166630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.166610 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:32.167838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.167816 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ba844c-openshift-default-dockercfg-xd85j\"" Nov 04 01:12:32.213392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213305 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213411 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213455 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213522 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5168ab-db6a-423c-81e8-80011c9d13f9-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213567 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213603 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hzql\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-kube-api-access-9hzql\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213633 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213670 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.213752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.213712 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.219980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.219954 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7"] Nov 04 01:12:32.221141 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:32.221116 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4e203f_a426_4905_a1a2_59c3bf5ce1ff.slice/crio-1652f7fc47138a50c3aed4e4cafee542cc9edb97387f4d3698fd6bda88ae9ef6 WatchSource:0}: Error finding container 1652f7fc47138a50c3aed4e4cafee542cc9edb97387f4d3698fd6bda88ae9ef6: Status 404 returned error can't find the container with id 1652f7fc47138a50c3aed4e4cafee542cc9edb97387f4d3698fd6bda88ae9ef6 Nov 04 01:12:32.223788 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.223757 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.223889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.223830 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.223889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.223860 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.322391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.293783 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.324730 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.324789 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.324816 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5168ab-db6a-423c-81e8-80011c9d13f9-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.324874 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.324910 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hzql\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-kube-api-access-9hzql\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.324942 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.324977 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.325033 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.325204 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.326025 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.326286 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.327000 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5168ab-db6a-423c-81e8-80011c9d13f9-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.331142 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.340701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.334087 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.360829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.348095 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.360829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.348298 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.360829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.354255 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.360829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.356926 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hzql\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-kube-api-access-9hzql\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.473898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.473788 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:32.478579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.477648 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv"] Nov 04 01:12:32.488052 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:32.488006 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638bc5b7_a66e_43d9_8053_b754b0b00afc.slice/crio-d5e32d8cfc1b7f7ae0d77e6cf3a7bce449436ee2707733b6abcbbb41c788d73f WatchSource:0}: Error finding container d5e32d8cfc1b7f7ae0d77e6cf3a7bce449436ee2707733b6abcbbb41c788d73f: Status 404 returned error can't find the container with id d5e32d8cfc1b7f7ae0d77e6cf3a7bce449436ee2707733b6abcbbb41c788d73f Nov 04 01:12:32.499412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.497591 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.499412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.497686 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.499412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.497730 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.676825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.676792 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt"] Nov 04 01:12:32.682977 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:32.682945 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298f1428_b1b5_40cd_851f_f3de8c0056a4.slice/crio-cf1a9610a103c8681a8d36f2c193728d033905f3e057f5b33f1c550b9c0480de WatchSource:0}: Error finding container cf1a9610a103c8681a8d36f2c193728d033905f3e057f5b33f1c550b9c0480de: Status 404 returned error can't find the container with id cf1a9610a103c8681a8d36f2c193728d033905f3e057f5b33f1c550b9c0480de Nov 04 01:12:32.686818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.685769 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.686942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.686918 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.687029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.686988 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.692417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.692329 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" event={"ID":"638bc5b7-a66e-43d9-8053-b754b0b00afc","Type":"ContainerStarted","Data":"4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0"} Nov 04 01:12:32.692417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.692394 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" event={"ID":"638bc5b7-a66e-43d9-8053-b754b0b00afc","Type":"ContainerStarted","Data":"d5e32d8cfc1b7f7ae0d77e6cf3a7bce449436ee2707733b6abcbbb41c788d73f"} Nov 04 01:12:32.694124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.694101 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" event={"ID":"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff","Type":"ContainerStarted","Data":"89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98"} Nov 04 01:12:32.694238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.694151 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" event={"ID":"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff","Type":"ContainerStarted","Data":"1652f7fc47138a50c3aed4e4cafee542cc9edb97387f4d3698fd6bda88ae9ef6"} Nov 04 01:12:32.696198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.695698 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" event={"ID":"a3359d44-f1ed-463c-a61e-52f2ce59795d","Type":"ContainerStarted","Data":"aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8"} Nov 04 01:12:32.696198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.695739 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" event={"ID":"a3359d44-f1ed-463c-a61e-52f2ce59795d","Type":"ContainerStarted","Data":"17db6f2485a924c882faac7e488008e5e4c51b2640cb1b0c55ca0c936b448072"} Nov 04 01:12:32.696934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.696906 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" event={"ID":"298f1428-b1b5-40cd-851f-f3de8c0056a4","Type":"ContainerStarted","Data":"cf1a9610a103c8681a8d36f2c193728d033905f3e057f5b33f1c550b9c0480de"} Nov 04 01:12:32.716382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.716297 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podStartSLOduration=1.716276808 podStartE2EDuration="1.716276808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:32.715015467 +0000 UTC m=+584.737947475" watchObservedRunningTime="2026-11-04 01:12:32.716276808 +0000 UTC m=+584.739208816" Nov 04 01:12:32.740168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.740034 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podStartSLOduration=1.740017415 podStartE2EDuration="1.740017415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:32.737864701 +0000 UTC m=+584.760796736" watchObservedRunningTime="2026-11-04 01:12:32.740017415 +0000 UTC m=+584.762949424" Nov 04 01:12:32.755141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.755114 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:32.765358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.765301 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podStartSLOduration=1.765287193 podStartE2EDuration="1.765287193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:32.762164975 +0000 UTC m=+584.785097001" watchObservedRunningTime="2026-11-04 01:12:32.765287193 +0000 UTC m=+584.788219199" Nov 04 01:12:32.784752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.784691 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5"] Nov 04 01:12:32.787471 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:32.787409 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5168ab_db6a_423c_81e8_80011c9d13f9.slice/crio-038613efe4eacfd4691f7608eb4a92d0fbaf0847b4ac3a93463e2a699a12aa91 WatchSource:0}: Error finding container 038613efe4eacfd4691f7608eb4a92d0fbaf0847b4ac3a93463e2a699a12aa91: Status 404 returned error can't find the container with id 038613efe4eacfd4691f7608eb4a92d0fbaf0847b4ac3a93463e2a699a12aa91 Nov 04 01:12:32.790561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.790525 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.790658 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.790629 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.790718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.790672 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:12:32.924639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:32.924604 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:33.168238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.168198 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:33.174478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.174447 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:33.703070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.703035 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" event={"ID":"dc5168ab-db6a-423c-81e8-80011c9d13f9","Type":"ContainerStarted","Data":"ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0"} Nov 04 01:12:33.703589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.703093 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" event={"ID":"dc5168ab-db6a-423c-81e8-80011c9d13f9","Type":"ContainerStarted","Data":"038613efe4eacfd4691f7608eb4a92d0fbaf0847b4ac3a93463e2a699a12aa91"} Nov 04 01:12:33.704659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.704636 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" event={"ID":"298f1428-b1b5-40cd-851f-f3de8c0056a4","Type":"ContainerStarted","Data":"e106d68eb8246c46cf36cca70120137ec48fc7b9c423cc8bdaa1590ae80de9c2"} Nov 04 01:12:33.705114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.705093 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:33.705658 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.705643 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:12:33.726983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.726937 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podStartSLOduration=1.726923798 podStartE2EDuration="1.726923798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:33.726160124 +0000 UTC m=+585.749092128" watchObservedRunningTime="2026-11-04 01:12:33.726923798 +0000 UTC m=+585.749855806" Nov 04 01:12:33.762547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.762521 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:33.779981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.779920 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podStartSLOduration=2.779902838 podStartE2EDuration="2.779902838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:33.776871683 +0000 UTC m=+585.799803718" watchObservedRunningTime="2026-11-04 01:12:33.779902838 +0000 UTC m=+585.802834847" Nov 04 01:12:33.934833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:33.934807 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:34.295511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.295479 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:34.301074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.301048 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:34.474740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.474710 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:34.480185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.480161 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:34.710823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.710743 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:34.710823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.710787 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:34.710823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.710805 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:34.710823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.710822 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:34.711453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.711426 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:12:34.711592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.711574 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:12:34.711659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.711625 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:12:34.711766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:34.711751 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:12:37.995473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:37.995440 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2"] Nov 04 01:12:37.999276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:37.999260 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.006649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.006630 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a5115a\"" Nov 04 01:12:38.009178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.009154 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2"] Nov 04 01:12:38.032090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.032064 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb"] Nov 04 01:12:38.035616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.035598 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.045792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.045770 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb"] Nov 04 01:12:38.104549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.104518 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a5115a-dockercfg-w24hp\"" Nov 04 01:12:38.123186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.123153 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjb9x\" (UniqueName: \"kubernetes.io/projected/461d2a93-be30-4606-932d-bc5c95097a64-kube-api-access-mjb9x\") pod \"payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.123501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.123477 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.123723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.123706 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzts\" (UniqueName: \"kubernetes.io/projected/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-kube-api-access-xwzts\") pod \"payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.123866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.123850 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461d2a93-be30-4606-932d-bc5c95097a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.222655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.222612 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752"] Nov 04 01:12:38.225587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.225557 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjb9x\" (UniqueName: \"kubernetes.io/projected/461d2a93-be30-4606-932d-bc5c95097a64-kube-api-access-mjb9x\") pod \"payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.225751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.225631 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.225751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.225696 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzts\" (UniqueName: \"kubernetes.io/projected/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-kube-api-access-xwzts\") pod \"payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.225751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.225739 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461d2a93-be30-4606-932d-bc5c95097a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.226727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.226699 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461d2a93-be30-4606-932d-bc5c95097a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.226979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.226952 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.227131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.227101 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.237874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.237136 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rbkg2\"" Nov 04 01:12:38.237874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.237204 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-9902c3-serving-cert\"" Nov 04 01:12:38.237874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.237861 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:12:38.238065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.237986 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:12:38.238521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.238496 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752"] Nov 04 01:12:38.246013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.245946 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzts\" (UniqueName: \"kubernetes.io/projected/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-kube-api-access-xwzts\") pod \"payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.249898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.249870 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjb9x\" (UniqueName: \"kubernetes.io/projected/461d2a93-be30-4606-932d-bc5c95097a64-kube-api-access-mjb9x\") pod \"payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.315839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.315816 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.327360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.327310 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-metrics-certs\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.327501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.327481 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.327557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.327543 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.327663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.327642 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfqx\" (UniqueName: \"kubernetes.io/projected/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-kube-api-access-4jfqx\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.333500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.333479 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:12:38.362477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.362350 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.429153 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-metrics-certs\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.429234 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.429265 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.429290 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfqx\" (UniqueName: \"kubernetes.io/projected/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-kube-api-access-4jfqx\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:12:38.429742 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-serving-cert: secret "maas-api-e2e-worker-w6-9902c3-serving-cert" not found Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:12:38.429816 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls podName:c3ae6bd1-8044-4ea8-8748-47afe26bf6eb nodeName:}" failed. No retries permitted until 2026-11-04 01:12:38.929795512 +0000 UTC m=+590.952727520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls") pod "maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" (UID: "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb") : secret "maas-api-e2e-worker-w6-9902c3-serving-cert" not found Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.430061 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.436745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.432301 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-metrics-certs\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.453320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.453077 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfqx\" (UniqueName: \"kubernetes.io/projected/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-kube-api-access-4jfqx\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.651918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.651888 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2"] Nov 04 01:12:38.654242 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:38.654189 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461d2a93_be30_4606_932d_bc5c95097a64.slice/crio-40eb9278afd7674279880bff9988c0df14b0b09a97e02dc06c587caead5f590a WatchSource:0}: Error finding container 40eb9278afd7674279880bff9988c0df14b0b09a97e02dc06c587caead5f590a: Status 404 returned error can't find the container with id 40eb9278afd7674279880bff9988c0df14b0b09a97e02dc06c587caead5f590a Nov 04 01:12:38.680933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.680907 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb"] Nov 04 01:12:38.686190 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:38.686160 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c87cbe9_94fa_479e_bb0d_38ad669eb71f.slice/crio-e44efac67bc59c33ef14380bbc25c902dfcc4260f03016203e9206b0f96cb691 WatchSource:0}: Error finding container e44efac67bc59c33ef14380bbc25c902dfcc4260f03016203e9206b0f96cb691: Status 404 returned error can't find the container with id e44efac67bc59c33ef14380bbc25c902dfcc4260f03016203e9206b0f96cb691 Nov 04 01:12:38.745955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.745927 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" event={"ID":"1c87cbe9-94fa-479e-bb0d-38ad669eb71f","Type":"ContainerStarted","Data":"e44efac67bc59c33ef14380bbc25c902dfcc4260f03016203e9206b0f96cb691"} Nov 04 01:12:38.747721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.747699 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" event={"ID":"461d2a93-be30-4606-932d-bc5c95097a64","Type":"ContainerStarted","Data":"ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173"} Nov 04 01:12:38.747844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.747744 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" event={"ID":"461d2a93-be30-4606-932d-bc5c95097a64","Type":"ContainerStarted","Data":"40eb9278afd7674279880bff9988c0df14b0b09a97e02dc06c587caead5f590a"} Nov 04 01:12:38.747906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.747848 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:38.774609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.774556 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" podStartSLOduration=1.774538275 podStartE2EDuration="1.774538275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:38.771990671 +0000 UTC m=+590.794922701" watchObservedRunningTime="2026-11-04 01:12:38.774538275 +0000 UTC m=+590.797470282" Nov 04 01:12:38.938354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.938249 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:38.940638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:38.940617 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls\") pod \"maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:39.147011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.146981 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:39.465955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.465927 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752"] Nov 04 01:12:39.468202 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:39.468175 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ae6bd1_8044_4ea8_8748_47afe26bf6eb.slice/crio-1463735d2790131e7d36ded6616580defde7c34f9494d58b404934a7265c4453 WatchSource:0}: Error finding container 1463735d2790131e7d36ded6616580defde7c34f9494d58b404934a7265c4453: Status 404 returned error can't find the container with id 1463735d2790131e7d36ded6616580defde7c34f9494d58b404934a7265c4453 Nov 04 01:12:39.756869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.754812 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" event={"ID":"1c87cbe9-94fa-479e-bb0d-38ad669eb71f","Type":"ContainerStarted","Data":"cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7"} Nov 04 01:12:39.756869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.754976 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:39.756869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.756760 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" event={"ID":"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb","Type":"ContainerStarted","Data":"1463735d2790131e7d36ded6616580defde7c34f9494d58b404934a7265c4453"} Nov 04 01:12:39.778068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.778008 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" podStartSLOduration=1.77798983 podStartE2EDuration="1.77798983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:39.776558614 +0000 UTC m=+591.799490635" watchObservedRunningTime="2026-11-04 01:12:39.77798983 +0000 UTC m=+591.800921838" Nov 04 01:12:39.784419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.784391 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp"] Nov 04 01:12:39.790099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.790081 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:39.795564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.795530 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp"] Nov 04 01:12:39.852224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.852189 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-875528\"" Nov 04 01:12:39.852418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.852275 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh5lp\" (UniqueName: \"kubernetes.io/projected/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-kube-api-access-rh5lp\") pod \"payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:39.852418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.852300 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-875528-dockercfg-2v4jh\"" Nov 04 01:12:39.852418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.852315 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:39.954353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.953413 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh5lp\" (UniqueName: \"kubernetes.io/projected/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-kube-api-access-rh5lp\") pod \"payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:39.954353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.953487 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:39.954591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.954543 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:39.971015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:39.969974 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh5lp\" (UniqueName: \"kubernetes.io/projected/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-kube-api-access-rh5lp\") pod \"payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:40.106316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.106272 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:40.401210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.401186 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp"] Nov 04 01:12:40.403304 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:40.403271 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08660fcb_c4ba_4d07_b709_2eb9d6fcf4ce.slice/crio-f89b7a56ca42c9bba742e7782cd8b976be46afa9292789cf0ffc40b65d4e1d9a WatchSource:0}: Error finding container f89b7a56ca42c9bba742e7782cd8b976be46afa9292789cf0ffc40b65d4e1d9a: Status 404 returned error can't find the container with id f89b7a56ca42c9bba742e7782cd8b976be46afa9292789cf0ffc40b65d4e1d9a Nov 04 01:12:40.763384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.763269 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" event={"ID":"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb","Type":"ContainerStarted","Data":"96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e"} Nov 04 01:12:40.763384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.763356 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:40.765319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.765287 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" event={"ID":"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce","Type":"ContainerStarted","Data":"95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f"} Nov 04 01:12:40.765459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.765359 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" event={"ID":"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce","Type":"ContainerStarted","Data":"f89b7a56ca42c9bba742e7782cd8b976be46afa9292789cf0ffc40b65d4e1d9a"} Nov 04 01:12:40.765459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.765435 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:40.784413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.784369 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" podStartSLOduration=2.253122791 podStartE2EDuration="2.784317263s" totalImagesPullingTime="531.194472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:40.783287842 +0000 UTC m=+592.806219868" watchObservedRunningTime="2026-11-04 01:12:40.784317263 +0000 UTC m=+592.807249272" Nov 04 01:12:40.804862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:40.804820 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" podStartSLOduration=1.8048070040000002 podStartE2EDuration="1.804807004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:40.802991867 +0000 UTC m=+592.825923897" watchObservedRunningTime="2026-11-04 01:12:40.804807004 +0000 UTC m=+592.827739012" Nov 04 01:12:42.526632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.526597 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt"] Nov 04 01:12:42.530568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.530542 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.539941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.539677 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6dac99-serving-cert\"" Nov 04 01:12:42.543601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.543576 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt"] Nov 04 01:12:42.596597 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.596557 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjl6\" (UniqueName: \"kubernetes.io/projected/371f9c50-6be6-401a-b489-68abffe8bc46-kube-api-access-lhjl6\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.596762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.596640 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-metrics-certs\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.596762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.596714 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.596879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.596768 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371f9c50-6be6-401a-b489-68abffe8bc46-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.698248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.698211 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjl6\" (UniqueName: \"kubernetes.io/projected/371f9c50-6be6-401a-b489-68abffe8bc46-kube-api-access-lhjl6\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.698410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.698271 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-metrics-certs\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.698410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.698302 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.698410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.698329 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371f9c50-6be6-401a-b489-68abffe8bc46-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.699023 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.699000 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371f9c50-6be6-401a-b489-68abffe8bc46-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.700707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.700681 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.700707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.700698 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-metrics-certs\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.710631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.710611 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjl6\" (UniqueName: \"kubernetes.io/projected/371f9c50-6be6-401a-b489-68abffe8bc46-kube-api-access-lhjl6\") pod \"maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:42.850992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:42.850961 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:43.132714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:43.132687 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt"] Nov 04 01:12:43.134162 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:12:43.134137 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371f9c50_6be6_401a_b489_68abffe8bc46.slice/crio-166f00174b3bbae583729b26a5fa441d1afde3e85e29ec3a4328eb16814f01d2 WatchSource:0}: Error finding container 166f00174b3bbae583729b26a5fa441d1afde3e85e29ec3a4328eb16814f01d2: Status 404 returned error can't find the container with id 166f00174b3bbae583729b26a5fa441d1afde3e85e29ec3a4328eb16814f01d2 Nov 04 01:12:43.794370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:43.794202 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" event={"ID":"371f9c50-6be6-401a-b489-68abffe8bc46","Type":"ContainerStarted","Data":"f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256"} Nov 04 01:12:43.794370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:43.794273 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" event={"ID":"371f9c50-6be6-401a-b489-68abffe8bc46","Type":"ContainerStarted","Data":"166f00174b3bbae583729b26a5fa441d1afde3e85e29ec3a4328eb16814f01d2"} Nov 04 01:12:43.794808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:43.794488 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:43.815011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:43.814970 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" podStartSLOduration=1.380950177 podStartE2EDuration="1.814955388s" totalImagesPullingTime="434.005211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.813575762 +0000 UTC m=+595.836507794" watchObservedRunningTime="2026-11-04 01:12:43.814955388 +0000 UTC m=+595.837887394" Nov 04 01:12:45.034279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.034248 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq"] Nov 04 01:12:45.038175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.038154 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.045391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.045370 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ba844c-serving-cert\"" Nov 04 01:12:45.078697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.078590 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq"] Nov 04 01:12:45.139644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.139612 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-metrics-certs\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.139787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.139722 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.139787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.139757 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdk5\" (UniqueName: \"kubernetes.io/projected/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-kube-api-access-5hdk5\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.139787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.139782 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.241167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.241134 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.241167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.241170 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdk5\" (UniqueName: \"kubernetes.io/projected/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-kube-api-access-5hdk5\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.241409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.241234 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.241505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.241483 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-metrics-certs\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.242028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.242006 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.243774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.243755 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.243861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.243852 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-metrics-certs\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.273983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.273954 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdk5\" (UniqueName: \"kubernetes.io/projected/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-kube-api-access-5hdk5\") pod \"maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.385738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.385664 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:45.678406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.676542 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq"] Nov 04 01:12:45.805518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:45.805478 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" event={"ID":"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f","Type":"ContainerStarted","Data":"ceb4bc5659bfdc537f6c649fa1f89755fb099438771f1e27eebec25f5394e63f"} Nov 04 01:12:46.778019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:46.777993 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:12:46.817030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:46.816997 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" event={"ID":"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f","Type":"ContainerStarted","Data":"b20f81b4bbc912f91b78662ef70160082d7aa6640944bedcf9e840037d9a6587"} Nov 04 01:12:46.817193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:46.817142 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:46.837277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:46.837195 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" podStartSLOduration=1.356848904 podStartE2EDuration="1.83717994s" totalImagesPullingTime="480.331036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:46.836789156 +0000 UTC m=+598.859721175" watchObservedRunningTime="2026-11-04 01:12:46.83717994 +0000 UTC m=+598.860111947" Nov 04 01:12:49.786579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:49.786543 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:12:49.828852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:49.827647 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:12:50.766359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:50.763439 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:12:50.766359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:50.763508 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:12:50.801358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:50.800567 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:12:51.811166 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:51.810613 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:12:52.851531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:12:52.851496 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:12:59.470246 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:12:59.470195 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:02.421411 4312 generic.go:356] "Generic (PLEG): container finished" podID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerID="aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8" exitCode=0 Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:02.421535 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" event={"ID":"a3359d44-f1ed-463c-a61e-52f2ce59795d","Type":"ContainerDied","Data":"aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:02.422122 4312 scope.go:122] "RemoveContainer" containerID="aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8" Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:02.423577 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:02.423642 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:02.423676 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.429875 4312 generic.go:356] "Generic (PLEG): container finished" podID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerID="ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0" exitCode=0 Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.429953 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" event={"ID":"dc5168ab-db6a-423c-81e8-80011c9d13f9","Type":"ContainerDied","Data":"ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.430465 4312 scope.go:122] "RemoveContainer" containerID="ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0" Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.432052 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.432132 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" event={"ID":"a3359d44-f1ed-463c-a61e-52f2ce59795d","Type":"ContainerStarted","Data":"45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.432133 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.432250 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.706431 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.706479 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.759652 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.759795 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:13:03.763631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:03.759840 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 04 01:13:04.439628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.439596 4312 generic.go:356] "Generic (PLEG): container finished" podID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerID="e106d68eb8246c46cf36cca70120137ec48fc7b9c423cc8bdaa1590ae80de9c2" exitCode=0 Nov 04 01:13:04.439850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.439670 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" event={"ID":"298f1428-b1b5-40cd-851f-f3de8c0056a4","Type":"ContainerDied","Data":"e106d68eb8246c46cf36cca70120137ec48fc7b9c423cc8bdaa1590ae80de9c2"} Nov 04 01:13:04.440303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.440288 4312 scope.go:122] "RemoveContainer" containerID="e106d68eb8246c46cf36cca70120137ec48fc7b9c423cc8bdaa1590ae80de9c2" Nov 04 01:13:04.441416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.441397 4312 generic.go:356] "Generic (PLEG): container finished" podID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerID="4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0" exitCode=0 Nov 04 01:13:04.441486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.441467 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" event={"ID":"638bc5b7-a66e-43d9-8053-b754b0b00afc","Type":"ContainerDied","Data":"4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0"} Nov 04 01:13:04.441753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.441725 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.441808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.441793 4312 scope.go:122] "RemoveContainer" containerID="4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0" Nov 04 01:13:04.441841 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.441806 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.442683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.442556 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.445004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.444972 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerID="89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98" exitCode=0 Nov 04 01:13:04.445793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.445160 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" event={"ID":"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff","Type":"ContainerDied","Data":"89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98"} Nov 04 01:13:04.446464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.446173 4312 scope.go:122] "RemoveContainer" containerID="89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98" Nov 04 01:13:04.469360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.448814 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.469360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.449291 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.469360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.449661 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.488222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.473139 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" event={"ID":"dc5168ab-db6a-423c-81e8-80011c9d13f9","Type":"ContainerStarted","Data":"7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf"} Nov 04 01:13:04.488222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.474287 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:13:04.504626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.488555 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:13:04.504626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.488643 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:13:04.525900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.525862 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.526033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.525942 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.526033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.525977 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:13:04.713703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.713676 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:13:04.719762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.718468 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:13:04.719762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.718511 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:13:04.719762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.718539 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:13:04.763353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.756738 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:13:04.763353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:04.756791 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 04 01:13:05.484368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.484315 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:13:05.498808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.498769 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:13:05.524207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.524155 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" event={"ID":"298f1428-b1b5-40cd-851f-f3de8c0056a4","Type":"ContainerStarted","Data":"2cc50ef75e73434cc37ce3d696cd93ddadff3c30fecaa2543c7a576df3b34aa0"} Nov 04 01:13:05.534799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.534751 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" event={"ID":"638bc5b7-a66e-43d9-8053-b754b0b00afc","Type":"ContainerStarted","Data":"b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128"} Nov 04 01:13:05.546589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.545980 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" event={"ID":"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff","Type":"ContainerStarted","Data":"6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385"} Nov 04 01:13:05.771859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.771767 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:13:05.772111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.772082 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 04 01:13:05.939597 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.939501 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:13:05.959360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.941438 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:05.959360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:05.941496 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:06.183005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.178004 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:06.183005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.178061 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:06.206360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.192847 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:13:06.296359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.295716 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:13:06.300356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.297749 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:06.300356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.297805 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:06.506414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.494705 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:13:06.506414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.494768 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:13:06.809387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.806065 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:13:06.809387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.806137 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 04 01:13:06.921787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.921747 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:06.922079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:06.922051 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:07.169845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.169756 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:07.170105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.170074 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:07.305881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.305843 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:07.306178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.306148 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:07.478960 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.478861 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:13:07.479285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.479236 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:13:07.759546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.759447 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:13:07.760136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.760102 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 04 01:13:07.924356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.923439 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:07.924356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:07.923518 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:08.206358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.199440 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:08.206358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.199496 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:08.296877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.296839 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:08.297174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.297130 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:08.477849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.477754 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:13:08.478099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.478068 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:13:08.756839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.756747 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:13:08.757092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.757064 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 04 01:13:08.921356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.920730 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:08.921356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:08.920796 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:09.179361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.178208 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:09.179361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.178269 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:09.295876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.295842 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:09.296159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.296122 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:09.475417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.475304 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:13:09.475697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.475668 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:13:09.787413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.764369 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 04 01:13:09.788168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.787693 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 04 01:13:09.924895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.924858 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:09.925553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:09.925508 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:10.182148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.169722 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:10.182148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.169779 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:10.299355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.298760 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:10.299355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.298826 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:10.474810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.474719 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:13:10.475078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.475046 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:13:10.817701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.817665 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:13:10.863360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.851675 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:13:10.926754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.926719 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:10.927351 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:10.927300 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:11.182515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:11.170325 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:11.182515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:11.170399 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:11.424490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:11.424443 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:11.424805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:11.424774 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:11.919837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:11.919801 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:11.920155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:11.920105 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:12.183474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.183375 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 04 01:13:12.184150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.184121 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 04 01:13:12.260715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.260686 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:13:12.283355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.282620 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:13:12.295800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.295769 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:12.296114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.296066 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:12.919002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.918965 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 04 01:13:12.919189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:12.919041 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 04 01:13:13.295631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.295593 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 04 01:13:13.296048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.295661 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 04 01:13:13.422296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.422260 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:13:13.423371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.423344 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:13:13.424989 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.424962 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:13:13.923392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.923364 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:13:13.923771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.923748 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:13:13.924328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:13.924309 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:13:14.342616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:14.342589 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:13:14.343069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:14.342894 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:13:14.344022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:14.343998 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:13:17.391413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.391380 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w"] Nov 04 01:13:17.396584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.396566 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.403219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.403184 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"kube-root-ca.crt\"" Nov 04 01:13:17.403410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.403288 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"openshift-service-ca.crt\"" Nov 04 01:13:17.403410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.403290 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"default-dockercfg-qvhgx\"" Nov 04 01:13:17.403547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.403408 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"premium-w2-cffa80-kserve-self-signed-certs\"" Nov 04 01:13:17.406748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.406719 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w"] Nov 04 01:13:17.453410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.453372 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-dshm\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.453600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.453427 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-tmp-dir\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.453600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.453508 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgc2\" (UniqueName: \"kubernetes.io/projected/ad993ef4-97b8-476b-a855-e3be3b893f94-kube-api-access-gkgc2\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.453600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.453592 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-model-cache\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.453712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.453629 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad993ef4-97b8-476b-a855-e3be3b893f94-tls-certs\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.453756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.453735 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-home\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.555712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.555670 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-home\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.556034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.556008 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-dshm\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.556194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.556176 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-tmp-dir\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.556316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.556299 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgc2\" (UniqueName: \"kubernetes.io/projected/ad993ef4-97b8-476b-a855-e3be3b893f94-kube-api-access-gkgc2\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.556503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.556485 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-model-cache\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.556638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.556624 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad993ef4-97b8-476b-a855-e3be3b893f94-tls-certs\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.556794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.556182 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-home\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.557802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.557129 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-tmp-dir\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.557947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.557732 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-model-cache\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.560655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.560634 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-dshm\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.561149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.561116 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad993ef4-97b8-476b-a855-e3be3b893f94-tls-certs\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.573361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.573285 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgc2\" (UniqueName: \"kubernetes.io/projected/ad993ef4-97b8-476b-a855-e3be3b893f94-kube-api-access-gkgc2\") pod \"premium-w2-cffa80-kserve-76db488df6-kdw2w\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.712054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.711963 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:17.988568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:17.988543 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w"] Nov 04 01:13:17.990418 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:13:17.990393 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad993ef4_97b8_476b_a855_e3be3b893f94.slice/crio-fffee58c28a8842e9a4e1b94ba55579e032cc6df3e709e1efe7be1917cdc6879 WatchSource:0}: Error finding container fffee58c28a8842e9a4e1b94ba55579e032cc6df3e709e1efe7be1917cdc6879: Status 404 returned error can't find the container with id fffee58c28a8842e9a4e1b94ba55579e032cc6df3e709e1efe7be1917cdc6879 Nov 04 01:13:18.937631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:18.937593 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" event={"ID":"ad993ef4-97b8-476b-a855-e3be3b893f94","Type":"ContainerStarted","Data":"98849059aff2d24761db53d96f0c3bda9ac2abbb8a6767b08d3efd95aa227b15"} Nov 04 01:13:18.938049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:18.937643 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" event={"ID":"ad993ef4-97b8-476b-a855-e3be3b893f94","Type":"ContainerStarted","Data":"fffee58c28a8842e9a4e1b94ba55579e032cc6df3e709e1efe7be1917cdc6879"} Nov 04 01:13:18.957014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:18.956956 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podStartSLOduration=1.9569437920000001 podStartE2EDuration="1.956943792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:18.956298568 +0000 UTC m=+630.979230597" watchObservedRunningTime="2026-11-04 01:13:18.956943792 +0000 UTC m=+630.979875798" Nov 04 01:13:19.712307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:19.712269 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:27.712289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:27.712236 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:27.725277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:27.725252 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:28.019760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:28.019671 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:13:29.473369 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:13:29.473314 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:13:30.311435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.311400 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg"] Nov 04 01:13:30.315545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.315506 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.324090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.324061 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg"] Nov 04 01:13:30.325276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.325201 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"kube-root-ca.crt\"" Nov 04 01:13:30.325429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.325302 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"openshift-service-ca.crt\"" Nov 04 01:13:30.325854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.325591 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"default-dockercfg-j4zvf\"" Nov 04 01:13:30.325854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.325724 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"premium-w0-6dac99-kserve-self-signed-certs\"" Nov 04 01:13:30.417208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.417172 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-tmp-dir\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.417381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.417224 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-dshm\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.417381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.417367 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-model-cache\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.417476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.417415 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-home\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.417476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.417434 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594998cc-00a8-4cdd-b3f8-9e29a915871a-tls-certs\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.417564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.417528 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg77q\" (UniqueName: \"kubernetes.io/projected/594998cc-00a8-4cdd-b3f8-9e29a915871a-kube-api-access-xg77q\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.518793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.518762 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-home\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.518793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.518794 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594998cc-00a8-4cdd-b3f8-9e29a915871a-tls-certs\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.519387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.518814 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg77q\" (UniqueName: \"kubernetes.io/projected/594998cc-00a8-4cdd-b3f8-9e29a915871a-kube-api-access-xg77q\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.519387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.518906 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-tmp-dir\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.519387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.518949 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-dshm\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.519387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.519035 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-model-cache\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.519387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.519166 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-home\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.519387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.519272 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-tmp-dir\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.519387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.519311 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-model-cache\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.521183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.521161 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-dshm\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.521319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.521301 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594998cc-00a8-4cdd-b3f8-9e29a915871a-tls-certs\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.532133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.532111 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg77q\" (UniqueName: \"kubernetes.io/projected/594998cc-00a8-4cdd-b3f8-9e29a915871a-kube-api-access-xg77q\") pod \"premium-w0-6dac99-kserve-9896df694-r8qgg\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.631127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.631055 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:30.916742 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:30.916714 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg"] Nov 04 01:13:30.918417 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:13:30.918391 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod594998cc_00a8_4cdd_b3f8_9e29a915871a.slice/crio-24d98aa809cd9f605f23b4d1eee8d8688fc75d0e4d26fc2fd5dba200750f8bfe WatchSource:0}: Error finding container 24d98aa809cd9f605f23b4d1eee8d8688fc75d0e4d26fc2fd5dba200750f8bfe: Status 404 returned error can't find the container with id 24d98aa809cd9f605f23b4d1eee8d8688fc75d0e4d26fc2fd5dba200750f8bfe Nov 04 01:13:31.026789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:31.026757 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" event={"ID":"594998cc-00a8-4cdd-b3f8-9e29a915871a","Type":"ContainerStarted","Data":"9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f"} Nov 04 01:13:31.026954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:31.026814 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" event={"ID":"594998cc-00a8-4cdd-b3f8-9e29a915871a","Type":"ContainerStarted","Data":"24d98aa809cd9f605f23b4d1eee8d8688fc75d0e4d26fc2fd5dba200750f8bfe"} Nov 04 01:13:31.046825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:31.046780 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podStartSLOduration=1.046760769 podStartE2EDuration="1.046760769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:31.045087246 +0000 UTC m=+643.068019273" watchObservedRunningTime="2026-11-04 01:13:31.046760769 +0000 UTC m=+643.069692773" Nov 04 01:13:31.631770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:31.631739 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:40.631583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:40.631553 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:40.644355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:40.644319 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:41.133273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:41.133244 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:13:42.037913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.037878 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n"] Nov 04 01:13:42.042250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.042220 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.059549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.059507 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"kube-root-ca.crt\"" Nov 04 01:13:42.059880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.059860 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"default-dockercfg-26cpk\"" Nov 04 01:13:42.060041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.060026 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"openshift-service-ca.crt\"" Nov 04 01:13:42.100368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.086810 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n"] Nov 04 01:13:42.112271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.089496 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"premium-w5-875528-kserve-self-signed-certs\"" Nov 04 01:13:42.209353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.201494 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-tmp-dir\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.209353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.201553 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8w2k\" (UniqueName: \"kubernetes.io/projected/90782a96-bee3-418b-ac21-b65ef47bfa10-kube-api-access-g8w2k\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.209353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.201596 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-home\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.209353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.201653 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90782a96-bee3-418b-ac21-b65ef47bfa10-tls-certs\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.209353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.201770 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-dshm\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.209353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.201809 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-model-cache\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.305411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.304908 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-dshm\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.305411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.304959 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-model-cache\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.305411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.305073 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-tmp-dir\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.305411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.305107 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8w2k\" (UniqueName: \"kubernetes.io/projected/90782a96-bee3-418b-ac21-b65ef47bfa10-kube-api-access-g8w2k\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.305411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.305147 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-home\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.305411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.305198 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90782a96-bee3-418b-ac21-b65ef47bfa10-tls-certs\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.317367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.307324 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-tmp-dir\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.317367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.307671 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-model-cache\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.317367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.307677 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-home\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.321353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.319139 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-dshm\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.336355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.328117 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90782a96-bee3-418b-ac21-b65ef47bfa10-tls-certs\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.337245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.336683 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8w2k\" (UniqueName: \"kubernetes.io/projected/90782a96-bee3-418b-ac21-b65ef47bfa10-kube-api-access-g8w2k\") pod \"premium-w5-875528-kserve-6f476df68b-jpt2n\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.514864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.514828 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:42.792090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:42.792067 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n"] Nov 04 01:13:42.793951 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:13:42.793924 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90782a96_bee3_418b_ac21_b65ef47bfa10.slice/crio-d76d4ac72ff743915b1e351526177d45e340955c3729f3284b66c4146e10a987 WatchSource:0}: Error finding container d76d4ac72ff743915b1e351526177d45e340955c3729f3284b66c4146e10a987: Status 404 returned error can't find the container with id d76d4ac72ff743915b1e351526177d45e340955c3729f3284b66c4146e10a987 Nov 04 01:13:43.247480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:43.246462 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" event={"ID":"90782a96-bee3-418b-ac21-b65ef47bfa10","Type":"ContainerStarted","Data":"61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c"} Nov 04 01:13:43.247480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:43.246516 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" event={"ID":"90782a96-bee3-418b-ac21-b65ef47bfa10","Type":"ContainerStarted","Data":"d76d4ac72ff743915b1e351526177d45e340955c3729f3284b66c4146e10a987"} Nov 04 01:13:43.268448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:43.268394 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podStartSLOduration=1.268376344 podStartE2EDuration="1.268376344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:43.266746142 +0000 UTC m=+655.289678150" watchObservedRunningTime="2026-11-04 01:13:43.268376344 +0000 UTC m=+655.291308353" Nov 04 01:13:43.515217 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:43.515126 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:52.515108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:52.515071 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:52.531191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:52.531165 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:53.335469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:13:53.335438 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:13:59.456972 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:13:59.456893 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:14:01.886416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.886379 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929"] Nov 04 01:14:01.890381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.890355 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.908758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.908731 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929"] Nov 04 01:14:01.914542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.914518 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6dac99-openshift-default-dockercfg-cxxq4\"" Nov 04 01:14:01.941126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941091 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941124 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941160 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941375 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2550f115-5ac3-4059-a467-83b06601cda5-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941457 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9t5x\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-kube-api-access-f9t5x\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941485 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941513 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941556 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2550f115-5ac3-4059-a467-83b06601cda5-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:01.941743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:01.941718 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.042640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042604 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.042807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042667 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.042807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042689 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.042807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042718 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.042807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042757 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2550f115-5ac3-4059-a467-83b06601cda5-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042806 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9t5x\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-kube-api-access-f9t5x\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042841 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042879 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.042903 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2550f115-5ac3-4059-a467-83b06601cda5-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.043145 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.043173 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.043538 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.043615 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.043903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.043880 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2550f115-5ac3-4059-a467-83b06601cda5-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.045043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.045011 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.045534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.045513 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2550f115-5ac3-4059-a467-83b06601cda5-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.055431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.055408 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.055655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.055639 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9t5x\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-kube-api-access-f9t5x\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.207402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.205738 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:02.536817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.536776 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929"] Nov 04 01:14:02.539605 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:14:02.539578 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2550f115_5ac3_4059_a467_83b06601cda5.slice/crio-6a64b7bddbedabc8b73ccfb1e6005ac052e24d3ca6ae97894c1d80a2faca7b70 WatchSource:0}: Error finding container 6a64b7bddbedabc8b73ccfb1e6005ac052e24d3ca6ae97894c1d80a2faca7b70: Status 404 returned error can't find the container with id 6a64b7bddbedabc8b73ccfb1e6005ac052e24d3ca6ae97894c1d80a2faca7b70 Nov 04 01:14:02.552078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.552017 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:14:02.552208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.552186 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:14:02.552313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:02.552288 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:14:03.383758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:03.383726 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" event={"ID":"2550f115-5ac3-4059-a467-83b06601cda5","Type":"ContainerStarted","Data":"3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0"} Nov 04 01:14:03.384160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:03.383785 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" event={"ID":"2550f115-5ac3-4059-a467-83b06601cda5","Type":"ContainerStarted","Data":"6a64b7bddbedabc8b73ccfb1e6005ac052e24d3ca6ae97894c1d80a2faca7b70"} Nov 04 01:14:03.408307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:03.408254 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podStartSLOduration=2.408238236 podStartE2EDuration="2.408238236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:03.405394552 +0000 UTC m=+675.428326579" watchObservedRunningTime="2026-11-04 01:14:03.408238236 +0000 UTC m=+675.431170323" Nov 04 01:14:04.206701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:04.206662 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:04.208056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:04.208031 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:14:04.208172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:04.208087 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 04 01:14:05.206708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:05.206679 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:14:05.207102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:05.206731 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 04 01:14:06.207373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:06.207312 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:14:06.207773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:06.207396 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 04 01:14:07.206990 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:07.206947 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:14:07.207217 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:07.207005 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 04 01:14:08.206778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:08.206748 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:14:08.207156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:08.206798 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 04 01:14:09.207617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:09.207578 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:14:09.208005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:09.207638 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 04 01:14:10.212539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:10.212511 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:10.212921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:10.212893 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:10.213606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:10.213585 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:14:17.217492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.217457 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd"] Nov 04 01:14:17.222563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.222543 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.235657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.235628 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-875528-openshift-default-dockercfg-9xdkz\"" Nov 04 01:14:17.237503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.236081 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd"] Nov 04 01:14:17.307977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.307934 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qw24\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-kube-api-access-5qw24\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.307977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.307972 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.308230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.308082 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.308230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.308115 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.308230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.308130 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.308230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.308154 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.308485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.308281 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.308485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.308314 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.308485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.308378 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31fac3-3297-4d81-9439-0fef57db1aa0-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.409420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409376 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qw24\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-kube-api-access-5qw24\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.409420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409430 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.409698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409496 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.409698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409534 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.409698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409557 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.409698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409599 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.409698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409626 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.410034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409713 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.410034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409775 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31fac3-3297-4d81-9439-0fef57db1aa0-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.410034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409934 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.410034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.409994 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.410260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.410122 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.410260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.410207 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.410558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.410540 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31fac3-3297-4d81-9439-0fef57db1aa0-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.411896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.411878 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.412043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.412026 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.421660 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.421631 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.421790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.421767 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qw24\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-kube-api-access-5qw24\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.545438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.545410 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:17.834241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.834217 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd"] Nov 04 01:14:17.836031 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:14:17.836004 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf31fac3_3297_4d81_9439_0fef57db1aa0.slice/crio-46d9c98e6b538a00d5f07fbec516e8ec998a51dcd55fe578f96c10968ce5db15 WatchSource:0}: Error finding container 46d9c98e6b538a00d5f07fbec516e8ec998a51dcd55fe578f96c10968ce5db15: Status 404 returned error can't find the container with id 46d9c98e6b538a00d5f07fbec516e8ec998a51dcd55fe578f96c10968ce5db15 Nov 04 01:14:17.838821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.838787 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:14:17.838905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.838865 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:14:17.838905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:17.838894 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:14:18.486480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:18.486433 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" event={"ID":"af31fac3-3297-4d81-9439-0fef57db1aa0","Type":"ContainerStarted","Data":"9dbeea53531ac47451b81cb904738b6dd5f15aed6bd43efc062373fc51c907a9"} Nov 04 01:14:18.486948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:18.486514 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" event={"ID":"af31fac3-3297-4d81-9439-0fef57db1aa0","Type":"ContainerStarted","Data":"46d9c98e6b538a00d5f07fbec516e8ec998a51dcd55fe578f96c10968ce5db15"} Nov 04 01:14:18.511726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:18.511676 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podStartSLOduration=1.511660614 podStartE2EDuration="1.511660614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:18.509426125 +0000 UTC m=+690.532358133" watchObservedRunningTime="2026-11-04 01:14:18.511660614 +0000 UTC m=+690.534592622" Nov 04 01:14:18.546433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:18.546406 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:18.583974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:18.583922 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:14:18.584278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:18.584241 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 04 01:14:19.557359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:19.556467 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:14:19.557359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:19.556543 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 04 01:14:20.545751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:20.545720 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:14:20.545957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:20.545774 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 04 01:14:21.546507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:21.546469 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:14:21.546921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:21.546527 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 04 01:14:22.546001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:22.545956 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:14:22.546171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:22.546019 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 04 01:14:23.546173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:23.546142 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:14:23.546684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:23.546196 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 04 01:14:24.550248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:24.550221 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:24.550720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:24.550597 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:24.551320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:24.551305 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:14:29.457687 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:14:29.457648 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:14:56.027633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.027600 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z"] Nov 04 01:14:56.032822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.032800 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.044614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.044578 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z"] Nov 04 01:14:56.045259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.045236 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:14:56.045502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.045487 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:14:56.106608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.106575 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.106959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.106934 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.107860 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.107837 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6g7k\" (UniqueName: \"kubernetes.io/projected/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-kube-api-access-p6g7k\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.209184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.209154 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.209373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.209251 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6g7k\" (UniqueName: \"kubernetes.io/projected/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-kube-api-access-p6g7k\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.209373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.209301 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.209915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.209894 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.211824 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.211800 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.221581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.221560 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6g7k\" (UniqueName: \"kubernetes.io/projected/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-kube-api-access-p6g7k\") pod \"maas-api-revoke-sub-b4a657aa-gdv9z\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.388887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.388807 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:14:56.663860 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.663830 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z"] Nov 04 01:14:56.665851 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:14:56.665819 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3fd24d_a34a_4d8e_b315_0870d6ae29f3.slice/crio-733be43a0415731d834f6a61849bd708978c638aafa51369f8b1494c66087de6 WatchSource:0}: Error finding container 733be43a0415731d834f6a61849bd708978c638aafa51369f8b1494c66087de6: Status 404 returned error can't find the container with id 733be43a0415731d834f6a61849bd708978c638aafa51369f8b1494c66087de6 Nov 04 01:14:56.667959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.667943 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:14:56.874081 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:14:56.874047 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" event={"ID":"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3","Type":"ContainerStarted","Data":"733be43a0415731d834f6a61849bd708978c638aafa51369f8b1494c66087de6"} Nov 04 01:14:59.461433 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:14:59.461395 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:15:00.149516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.149482 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64"] Nov 04 01:15:00.153289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.153268 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.164724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.164692 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64"] Nov 04 01:15:00.172428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.172401 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv"] Nov 04 01:15:00.176951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.176930 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.186045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.186023 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv"] Nov 04 01:15:00.310358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.272107 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8xt\" (UniqueName: \"kubernetes.io/projected/2561425d-ce5b-4c4a-8d35-08ce31a3742d-kube-api-access-8w8xt\") pod \"maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.310358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.272255 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561425d-ce5b-4c4a-8d35-08ce31a3742d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.373124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.373088 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601694d3-057f-4f6a-932a-da07c973acf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.373324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.373250 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8xt\" (UniqueName: \"kubernetes.io/projected/2561425d-ce5b-4c4a-8d35-08ce31a3742d-kube-api-access-8w8xt\") pod \"maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.373445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.373377 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clxhx\" (UniqueName: \"kubernetes.io/projected/601694d3-057f-4f6a-932a-da07c973acf8-kube-api-access-clxhx\") pod \"maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.373508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.373482 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561425d-ce5b-4c4a-8d35-08ce31a3742d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.374243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.374220 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561425d-ce5b-4c4a-8d35-08ce31a3742d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.387044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.387010 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8xt\" (UniqueName: \"kubernetes.io/projected/2561425d-ce5b-4c4a-8d35-08ce31a3742d-kube-api-access-8w8xt\") pod \"maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.474581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.474547 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601694d3-057f-4f6a-932a-da07c973acf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.475010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.474649 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clxhx\" (UniqueName: \"kubernetes.io/projected/601694d3-057f-4f6a-932a-da07c973acf8-kube-api-access-clxhx\") pod \"maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.475279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.475253 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601694d3-057f-4f6a-932a-da07c973acf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.476280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.476262 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:00.488076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.488052 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clxhx\" (UniqueName: \"kubernetes.io/projected/601694d3-057f-4f6a-932a-da07c973acf8-kube-api-access-clxhx\") pod \"maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.525959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.525923 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:00.581381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.580950 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64"] Nov 04 01:15:00.590538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.587087 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.590538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.589567 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64"] Nov 04 01:15:00.678274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.678235 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtdj\" (UniqueName: \"kubernetes.io/projected/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-kube-api-access-vxtdj\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.678450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.678324 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.678498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.678438 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.786364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.784195 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.786364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.784281 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.786364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.784515 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtdj\" (UniqueName: \"kubernetes.io/projected/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-kube-api-access-vxtdj\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.786364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.785565 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.818894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.813005 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtdj\" (UniqueName: \"kubernetes.io/projected/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-kube-api-access-vxtdj\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.838370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.836932 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a464071e-ktt64\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.894359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.874494 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64"] Nov 04 01:15:00.919358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.919241 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv"] Nov 04 01:15:00.924570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.924537 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:00.970216 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.969008 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" event={"ID":"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3","Type":"ContainerStarted","Data":"83081907dda0291bf099a8ae7be45449afae08fc8fa1458a2291ca52236677db"} Nov 04 01:15:00.982643 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:00.982611 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerStarted","Data":"6fe29e5561fbf8c2315ebad452709bd51daa86323e51eded1b6271208a59e0b0"} Nov 04 01:15:01.003245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:01.003184 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" podStartSLOduration=1.509385173 podStartE2EDuration="5.003163301s" totalImagesPullingTime="3.493778128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:00.992589647 +0000 UTC m=+733.015521674" watchObservedRunningTime="2026-11-04 01:15:01.003163301 +0000 UTC m=+733.026095309" Nov 04 01:15:01.053322 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:01.053224 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601694d3_057f_4f6a_932a_da07c973acf8.slice/crio-d98573a489dd5e4ca978a204005b6ed12d2fea0e529a7915eab5cb7671c20ae1 WatchSource:0}: Error finding container d98573a489dd5e4ca978a204005b6ed12d2fea0e529a7915eab5cb7671c20ae1: Status 404 returned error can't find the container with id d98573a489dd5e4ca978a204005b6ed12d2fea0e529a7915eab5cb7671c20ae1 Nov 04 01:15:01.914308 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:01.914250 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857"] Nov 04 01:15:02.056186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.053731 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857"] Nov 04 01:15:02.056186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.053859 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.099363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099166 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.099363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099227 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.100257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099609 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.100257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099708 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.100257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099735 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.100257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099786 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.100257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099835 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.100257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099862 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.100257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.099946 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmwg\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-kube-api-access-6bmwg\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.247972 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248014 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248069 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248114 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248141 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248203 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmwg\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-kube-api-access-6bmwg\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248381 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248409 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.248445 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.249004 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.259201 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.259935 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.260357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.260196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.261191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.260464 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.610291 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.610257 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.613037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.613004 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.613767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.613741 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.641393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.640915 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmwg\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-kube-api-access-6bmwg\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.707876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.707793 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64"] Nov 04 01:15:02.709514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.709490 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg"] Nov 04 01:15:02.793355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.780754 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:02.925653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.922933 4312 generic.go:356] "Generic (PLEG): container finished" podID="3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" containerID="83081907dda0291bf099a8ae7be45449afae08fc8fa1458a2291ca52236677db" exitCode=0 Nov 04 01:15:02.925653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.924610 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg"] Nov 04 01:15:02.925653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.924647 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv"] Nov 04 01:15:02.936348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.926454 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:02.936348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.928190 4312 scope.go:122] "RemoveContainer" containerID="13b09d78ca946ca5c321e13a577c88c963c1d24ee437564dc545f2bfac2ea4b5" Nov 04 01:15:02.936348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.934202 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv"] Nov 04 01:15:02.936348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.934242 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" event={"ID":"601694d3-057f-4f6a-932a-da07c973acf8","Type":"ContainerStarted","Data":"d98573a489dd5e4ca978a204005b6ed12d2fea0e529a7915eab5cb7671c20ae1"} Nov 04 01:15:02.936348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.934272 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" event={"ID":"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3","Type":"ContainerDied","Data":"83081907dda0291bf099a8ae7be45449afae08fc8fa1458a2291ca52236677db"} Nov 04 01:15:02.936348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.934305 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerStarted","Data":"13b09d78ca946ca5c321e13a577c88c963c1d24ee437564dc545f2bfac2ea4b5"} Nov 04 01:15:02.936348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:02.934443 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.143577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143121 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.143577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143207 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.143577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143243 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.143577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143299 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5ed29b-ec80-4bef-bc71-4c185119193a-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.143577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143359 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwf7\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-kube-api-access-xlwf7\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.143577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143392 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.143577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143431 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.143475 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9j9z\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-kube-api-access-w9j9z\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144184 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481d9419-e346-444c-b319-a38160b62de7-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144221 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144246 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144287 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144352 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144381 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481d9419-e346-444c-b319-a38160b62de7-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144452 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144529 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144596 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.144622 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.245279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245244 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245292 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.245446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245322 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5ed29b-ec80-4bef-bc71-4c185119193a-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.245446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245371 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwf7\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-kube-api-access-xlwf7\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.245446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245391 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245408 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245445 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9j9z\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-kube-api-access-w9j9z\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245472 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481d9419-e346-444c-b319-a38160b62de7-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245503 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245526 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245559 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245585 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245612 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481d9419-e346-444c-b319-a38160b62de7-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245641 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245723 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.245803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245776 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.246312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245848 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.246312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245918 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.246312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245948 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.246312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.245971 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.246312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.246074 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.246312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.246279 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.247177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.246390 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5ed29b-ec80-4bef-bc71-4c185119193a-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.247177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.246971 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.247177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.246974 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.247413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.247392 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.248072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.248047 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481d9419-e346-444c-b319-a38160b62de7-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.251710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.251681 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.252261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.252237 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.253325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.253281 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.259230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.253916 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481d9419-e346-444c-b319-a38160b62de7-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.260354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.259588 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.264096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.264047 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.266476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.266432 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9j9z\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-kube-api-access-w9j9z\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.272838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.272792 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.275539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.275453 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwf7\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-kube-api-access-xlwf7\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.367246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.367211 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:03.376845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.376665 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:03.433788 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.433655 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857"] Nov 04 01:15:03.438286 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:03.438251 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15c9f63_0121_426d_b7ee_7bbae3e8ad82.slice/crio-1ac7ea23f4d183602272c4615f494e1c02da240303c138148237b7991c300d83 WatchSource:0}: Error finding container 1ac7ea23f4d183602272c4615f494e1c02da240303c138148237b7991c300d83: Status 404 returned error can't find the container with id 1ac7ea23f4d183602272c4615f494e1c02da240303c138148237b7991c300d83 Nov 04 01:15:03.441171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.441128 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.441247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.441234 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.441298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.441277 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.834405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.834376 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg"] Nov 04 01:15:03.837271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.837245 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv"] Nov 04 01:15:03.838885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.838775 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.838885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.838845 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.839415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.839055 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.841546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.841516 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.841727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.841708 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.841888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.841869 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:03.938446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.938409 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" event={"ID":"481d9419-e346-444c-b319-a38160b62de7","Type":"ContainerStarted","Data":"316adb800f62452a5bc32bc9799d39b2b3a13936c0927cda179e8486134eaecf"} Nov 04 01:15:03.941723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.941689 4312 generic.go:356] "Generic (PLEG): container finished" podID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerID="13b09d78ca946ca5c321e13a577c88c963c1d24ee437564dc545f2bfac2ea4b5" exitCode=60 Nov 04 01:15:03.941862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.941772 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerDied","Data":"13b09d78ca946ca5c321e13a577c88c963c1d24ee437564dc545f2bfac2ea4b5"} Nov 04 01:15:03.941862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.941811 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerStarted","Data":"a4a8b8732b481911322921586b6d4ef268fa86f4c499466ab231d6ade59625a1"} Nov 04 01:15:03.943579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.943526 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" event={"ID":"ee5ed29b-ec80-4bef-bc71-4c185119193a","Type":"ContainerStarted","Data":"41fff653bc9b86f2ca8abc5ee20955dc40d46fc3f5c4c20d1242c0713eb8a592"} Nov 04 01:15:03.944809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.944786 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" event={"ID":"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5","Type":"ContainerStarted","Data":"40e4979a8f0cb6d5e4001f3fee39c97540db4031dd6b6f3c9fb4f0c9edf1185e"} Nov 04 01:15:03.944930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.944823 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" event={"ID":"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5","Type":"ContainerStarted","Data":"a44bd6bfcf3e1789a12c73af10a67c5be21b8e18c991e74af1b6cd89b290134d"} Nov 04 01:15:03.946421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.946393 4312 generic.go:356] "Generic (PLEG): container finished" podID="601694d3-057f-4f6a-932a-da07c973acf8" containerID="d66303e901d0c0722d81db7fb262e36ca0c5921d349f6fcaee87a70fb4345692" exitCode=60 Nov 04 01:15:03.946539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.946462 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" event={"ID":"601694d3-057f-4f6a-932a-da07c973acf8","Type":"ContainerDied","Data":"d66303e901d0c0722d81db7fb262e36ca0c5921d349f6fcaee87a70fb4345692"} Nov 04 01:15:03.956362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.949436 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" event={"ID":"e15c9f63-0121-426d-b7ee-7bbae3e8ad82","Type":"ContainerStarted","Data":"5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640"} Nov 04 01:15:03.956362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.949484 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" event={"ID":"e15c9f63-0121-426d-b7ee-7bbae3e8ad82","Type":"ContainerStarted","Data":"1ac7ea23f4d183602272c4615f494e1c02da240303c138148237b7991c300d83"} Nov 04 01:15:03.978487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.978435 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" podStartSLOduration=3.978417488 podStartE2EDuration="3.978417488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:03.960634515 +0000 UTC m=+735.983566522" watchObservedRunningTime="2026-11-04 01:15:03.978417488 +0000 UTC m=+736.001349495" Nov 04 01:15:03.979585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.979562 4312 scope.go:122] "RemoveContainer" containerID="d66303e901d0c0722d81db7fb262e36ca0c5921d349f6fcaee87a70fb4345692" Nov 04 01:15:04.000500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:03.995310 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" podStartSLOduration=3.995292073 podStartE2EDuration="3.995292073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:03.992671563 +0000 UTC m=+736.015603571" watchObservedRunningTime="2026-11-04 01:15:03.995292073 +0000 UTC m=+736.018224084" Nov 04 01:15:04.038480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.029852 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podStartSLOduration=3.02983362 podStartE2EDuration="3.02983362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:04.027749141 +0000 UTC m=+736.050681160" watchObservedRunningTime="2026-11-04 01:15:04.02983362 +0000 UTC m=+736.052765628" Nov 04 01:15:04.779854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.779828 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:15:04.826592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.826430 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:04.889380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.889053 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-xks-ca\") pod \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " Nov 04 01:15:04.889380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.889110 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-openshift-ca\") pod \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " Nov 04 01:15:04.889380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.889174 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-kube-api-access-p6g7k\" (UniqueName: \"kubernetes.io/projected/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-kube-api-access-p6g7k\") pod \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\" (UID: \"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3\") " Nov 04 01:15:04.891723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.891693 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-openshift-ca" pod "3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" (UID: "3f3fd24d-a34a-4d8e-b315-0870d6ae29f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:04.892448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.892426 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:04.900588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.899457 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:15:04.900588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:04.899511 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:15:05.006069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.006024 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-kube-api-access-p6g7k" pod "3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" (UID: "3f3fd24d-a34a-4d8e-b315-0870d6ae29f3"). InnerVolumeSpecName "kube-api-access-p6g7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:05.007175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.007144 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6g7k\" (UniqueName: \"kubernetes.io/projected/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-kube-api-access-p6g7k\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:05.023681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.023483 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-xks-ca" pod "3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" (UID: "3f3fd24d-a34a-4d8e-b315-0870d6ae29f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:05.039609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.037971 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" event={"ID":"481d9419-e346-444c-b319-a38160b62de7","Type":"ContainerStarted","Data":"543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6"} Nov 04 01:15:05.042524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.042462 4312 generic.go:356] "Generic (PLEG): container finished" podID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerID="a4a8b8732b481911322921586b6d4ef268fa86f4c499466ab231d6ade59625a1" exitCode=60 Nov 04 01:15:05.043610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.042794 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerDied","Data":"a4a8b8732b481911322921586b6d4ef268fa86f4c499466ab231d6ade59625a1"} Nov 04 01:15:05.043610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.042857 4312 scope.go:122] "RemoveContainer" containerID="13b09d78ca946ca5c321e13a577c88c963c1d24ee437564dc545f2bfac2ea4b5" Nov 04 01:15:05.043610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.043316 4312 scope.go:122] "RemoveContainer" containerID="a4a8b8732b481911322921586b6d4ef268fa86f4c499466ab231d6ade59625a1" Nov 04 01:15:05.048435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.048021 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" event={"ID":"ee5ed29b-ec80-4bef-bc71-4c185119193a","Type":"ContainerStarted","Data":"37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09"} Nov 04 01:15:05.051205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.050943 4312 generic.go:356] "Generic (PLEG): container finished" podID="1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" containerID="40e4979a8f0cb6d5e4001f3fee39c97540db4031dd6b6f3c9fb4f0c9edf1185e" exitCode=0 Nov 04 01:15:05.051205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.051049 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" event={"ID":"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5","Type":"ContainerDied","Data":"40e4979a8f0cb6d5e4001f3fee39c97540db4031dd6b6f3c9fb4f0c9edf1185e"} Nov 04 01:15:05.054355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.054263 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" event={"ID":"601694d3-057f-4f6a-932a-da07c973acf8","Type":"ContainerStarted","Data":"2953e4371e5118d59deda85ba1be6cfe00d8083928eaa3215bcb06c49e2b9993"} Nov 04 01:15:05.060380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.057875 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" Nov 04 01:15:05.060713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.060555 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z" event={"ID":"3f3fd24d-a34a-4d8e-b315-0870d6ae29f3","Type":"ContainerDied","Data":"733be43a0415731d834f6a61849bd708978c638aafa51369f8b1494c66087de6"} Nov 04 01:15:05.060713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.060604 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733be43a0415731d834f6a61849bd708978c638aafa51369f8b1494c66087de6" Nov 04 01:15:05.061284 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:05.061021 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64_odh-ai-gateway-infra(2561425d-ce5b-4c4a-8d35-08ce31a3742d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" Nov 04 01:15:05.127881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.121729 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:05.146980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.135180 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podStartSLOduration=3.135160983 podStartE2EDuration="3.135160983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:05.082165989 +0000 UTC m=+737.105098014" watchObservedRunningTime="2026-11-04 01:15:05.135160983 +0000 UTC m=+737.158092993" Nov 04 01:15:05.291584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.284622 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podStartSLOduration=3.284596005 podStartE2EDuration="3.284596005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:05.227993785 +0000 UTC m=+737.250925792" watchObservedRunningTime="2026-11-04 01:15:05.284596005 +0000 UTC m=+737.307528013" Nov 04 01:15:05.311394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.309767 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" podStartSLOduration=5.293503985 podStartE2EDuration="5.293503985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:05.256821078 +0000 UTC m=+737.279753086" watchObservedRunningTime="2026-11-04 01:15:05.293503985 +0000 UTC m=+737.316435993" Nov 04 01:15:05.371358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.370766 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:05.371358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.370899 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:15:05.371358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.370945 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:15:05.385418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.384720 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:05.385418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.384846 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 04 01:15:05.385418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.384892 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 04 01:15:05.804245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.803781 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:15:05.804245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:05.803847 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:15:06.142371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.142256 4312 scope.go:122] "RemoveContainer" containerID="a4a8b8732b481911322921586b6d4ef268fa86f4c499466ab231d6ade59625a1" Nov 04 01:15:06.142826 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:06.142797 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64_odh-ai-gateway-infra(2561425d-ce5b-4c4a-8d35-08ce31a3742d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" Nov 04 01:15:06.144475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.144434 4312 generic.go:356] "Generic (PLEG): container finished" podID="601694d3-057f-4f6a-932a-da07c973acf8" containerID="2953e4371e5118d59deda85ba1be6cfe00d8083928eaa3215bcb06c49e2b9993" exitCode=60 Nov 04 01:15:06.145662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.145644 4312 scope.go:122] "RemoveContainer" containerID="2953e4371e5118d59deda85ba1be6cfe00d8083928eaa3215bcb06c49e2b9993" Nov 04 01:15:06.145939 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:06.145916 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv_odh-ai-gateway-infra(601694d3-057f-4f6a-932a-da07c973acf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" podUID="601694d3-057f-4f6a-932a-da07c973acf8" Nov 04 01:15:06.171209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.171164 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" event={"ID":"601694d3-057f-4f6a-932a-da07c973acf8","Type":"ContainerDied","Data":"2953e4371e5118d59deda85ba1be6cfe00d8083928eaa3215bcb06c49e2b9993"} Nov 04 01:15:06.171485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.171465 4312 scope.go:122] "RemoveContainer" containerID="d66303e901d0c0722d81db7fb262e36ca0c5921d349f6fcaee87a70fb4345692" Nov 04 01:15:06.373735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.372746 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:15:06.373735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.372811 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:15:06.390859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.390823 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 04 01:15:06.394402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.391075 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 04 01:15:06.822278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.822241 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:15:06.822573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.822543 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:15:06.988359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:06.980548 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:07.031987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.031948 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-openshift-ca\") pod \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " Nov 04 01:15:07.032181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.032045 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-kube-api-access-vxtdj\" (UniqueName: \"kubernetes.io/projected/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-kube-api-access-vxtdj\") pod \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " Nov 04 01:15:07.032181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.032088 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-xks-ca\") pod \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\" (UID: \"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5\") " Nov 04 01:15:07.034825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.034790 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-openshift-ca" pod "1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" (UID: "1bd44ceb-72d0-4d89-899c-7ef158d1a9a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:07.040702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.040672 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-xks-ca" pod "1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" (UID: "1bd44ceb-72d0-4d89-899c-7ef158d1a9a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:07.040817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.040800 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-kube-api-access-vxtdj" pod "1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" (UID: "1bd44ceb-72d0-4d89-899c-7ef158d1a9a5"). InnerVolumeSpecName "kube-api-access-vxtdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:07.140245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.140115 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtdj\" (UniqueName: \"kubernetes.io/projected/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-kube-api-access-vxtdj\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:07.140245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.140149 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:07.140245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.140163 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:07.236860 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.236825 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" event={"ID":"1bd44ceb-72d0-4d89-899c-7ef158d1a9a5","Type":"ContainerDied","Data":"a44bd6bfcf3e1789a12c73af10a67c5be21b8e18c991e74af1b6cd89b290134d"} Nov 04 01:15:07.237410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.236874 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44bd6bfcf3e1789a12c73af10a67c5be21b8e18c991e74af1b6cd89b290134d" Nov 04 01:15:07.237410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.236931 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64" Nov 04 01:15:07.240806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.240778 4312 scope.go:122] "RemoveContainer" containerID="2953e4371e5118d59deda85ba1be6cfe00d8083928eaa3215bcb06c49e2b9993" Nov 04 01:15:07.241000 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:07.240983 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv_odh-ai-gateway-infra(601694d3-057f-4f6a-932a-da07c973acf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" podUID="601694d3-057f-4f6a-932a-da07c973acf8" Nov 04 01:15:07.371900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.371829 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:15:07.372202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.372150 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:15:07.381262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.381235 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 04 01:15:07.381507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.381482 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 04 01:15:07.802358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.799731 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:15:07.802358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:07.799793 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:15:08.370360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:08.368699 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:15:08.370360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:08.368759 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:15:08.384499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:08.384466 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 04 01:15:08.384755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:08.384722 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 04 01:15:08.799368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:08.797734 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:15:08.799368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:08.797798 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:15:09.370355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:09.369791 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:15:09.370355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:09.369867 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:15:09.385356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:09.384475 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 04 01:15:09.385356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:09.384552 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 04 01:15:09.801355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:09.796679 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:15:09.801355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:09.796740 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:15:10.371760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:10.371719 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:15:10.372378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:10.372324 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:15:10.378677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:10.378650 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 04 01:15:10.378857 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:10.378833 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 04 01:15:10.815360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:10.814485 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:15:10.815360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:10.814551 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:15:11.225059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.224974 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr"] Nov 04 01:15:11.225952 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.225927 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" containerName="revoke-keys" Nov 04 01:15:11.225952 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.225953 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" containerName="revoke-keys" Nov 04 01:15:11.226095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.225982 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" containerName="revoke-keys" Nov 04 01:15:11.226095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.225990 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" containerName="revoke-keys" Nov 04 01:15:11.226156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.226132 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" containerName="revoke-keys" Nov 04 01:15:11.226156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.226148 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" containerName="revoke-keys" Nov 04 01:15:11.230384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.230362 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.234851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.234426 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:15:11.239354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.237741 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr"] Nov 04 01:15:11.339661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.339612 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.339848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.339717 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkpk\" (UniqueName: \"kubernetes.io/projected/c5a33c19-2e92-4037-b663-90074e2ea9f0-kube-api-access-6rkpk\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.339912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.339881 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.368098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.368061 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:15:11.368256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.368140 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:15:11.378415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.378386 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 04 01:15:11.378963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.378935 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 04 01:15:11.444286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.444245 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.444704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.444678 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.444854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.444834 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkpk\" (UniqueName: \"kubernetes.io/projected/c5a33c19-2e92-4037-b663-90074e2ea9f0-kube-api-access-6rkpk\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.446632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.446204 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.455439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.451063 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.463290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.463228 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkpk\" (UniqueName: \"kubernetes.io/projected/c5a33c19-2e92-4037-b663-90074e2ea9f0-kube-api-access-6rkpk\") pod \"maas-api-revoke-sub-fc288452-9wcqr\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.561587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.561551 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:11.824214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.824099 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:11.832856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.832485 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:11.835873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.835842 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:15:11.956513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:11.956480 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr"] Nov 04 01:15:12.326281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:12.326239 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" event={"ID":"c5a33c19-2e92-4037-b663-90074e2ea9f0","Type":"ContainerStarted","Data":"e7f7280375ef7ff91ca44292df6d75fe3210b6b6f2df459f9ea4e00a98a32378"} Nov 04 01:15:12.372468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:12.372444 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:12.372816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:12.372799 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:12.373429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:12.373410 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:15:12.380902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:12.380881 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:12.381167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:12.381136 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:12.381821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:12.381805 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:15:13.335163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:13.335079 4312 generic.go:356] "Generic (PLEG): container finished" podID="c5a33c19-2e92-4037-b663-90074e2ea9f0" containerID="00376732d643b0bb7712f672e0ba3268732aac61d75e9160766a6641b8e822d5" exitCode=0 Nov 04 01:15:13.335323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:13.335162 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" event={"ID":"c5a33c19-2e92-4037-b663-90074e2ea9f0","Type":"ContainerDied","Data":"00376732d643b0bb7712f672e0ba3268732aac61d75e9160766a6641b8e822d5"} Nov 04 01:15:14.198360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:14.195115 4312 scope.go:122] "RemoveContainer" containerID="a4a8b8732b481911322921586b6d4ef268fa86f4c499466ab231d6ade59625a1" Nov 04 01:15:15.218392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.206718 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:15.331745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.331704 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a33c19-2e92-4037-b663-90074e2ea9f0-kube-api-access-6rkpk\" (UniqueName: \"kubernetes.io/projected/c5a33c19-2e92-4037-b663-90074e2ea9f0-kube-api-access-6rkpk\") pod \"c5a33c19-2e92-4037-b663-90074e2ea9f0\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " Nov 04 01:15:15.349568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.349531 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-xks-ca\") pod \"c5a33c19-2e92-4037-b663-90074e2ea9f0\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " Nov 04 01:15:15.365358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.349912 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-openshift-ca\") pod \"c5a33c19-2e92-4037-b663-90074e2ea9f0\" (UID: \"c5a33c19-2e92-4037-b663-90074e2ea9f0\") " Nov 04 01:15:15.383419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.361550 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-openshift-ca" pod "c5a33c19-2e92-4037-b663-90074e2ea9f0" (UID: "c5a33c19-2e92-4037-b663-90074e2ea9f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:15.408677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.397450 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a33c19-2e92-4037-b663-90074e2ea9f0-kube-api-access-6rkpk" pod "c5a33c19-2e92-4037-b663-90074e2ea9f0" (UID: "c5a33c19-2e92-4037-b663-90074e2ea9f0"). InnerVolumeSpecName "kube-api-access-6rkpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:15.419815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.419760 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-xks-ca" pod "c5a33c19-2e92-4037-b663-90074e2ea9f0" (UID: "c5a33c19-2e92-4037-b663-90074e2ea9f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:15.483359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.465192 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" event={"ID":"c5a33c19-2e92-4037-b663-90074e2ea9f0","Type":"ContainerDied","Data":"e7f7280375ef7ff91ca44292df6d75fe3210b6b6f2df459f9ea4e00a98a32378"} Nov 04 01:15:15.483359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.465261 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f7280375ef7ff91ca44292df6d75fe3210b6b6f2df459f9ea4e00a98a32378" Nov 04 01:15:15.483359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.465362 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr" Nov 04 01:15:15.483359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.478298 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.483359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.478347 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a33c19-2e92-4037-b663-90074e2ea9f0-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.483359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.478367 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkpk\" (UniqueName: \"kubernetes.io/projected/c5a33c19-2e92-4037-b663-90074e2ea9f0-kube-api-access-6rkpk\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.483359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:15.481656 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerStarted","Data":"1c375458addc503cfc4c00c177d94920a0248c41d9b5ac534d69b1aef19f1438"} Nov 04 01:15:16.108634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:16.108593 4312 scope.go:122] "RemoveContainer" containerID="2953e4371e5118d59deda85ba1be6cfe00d8083928eaa3215bcb06c49e2b9993" Nov 04 01:15:16.489735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:16.489703 4312 generic.go:356] "Generic (PLEG): container finished" podID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerID="1c375458addc503cfc4c00c177d94920a0248c41d9b5ac534d69b1aef19f1438" exitCode=60 Nov 04 01:15:16.490156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:16.489779 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerDied","Data":"1c375458addc503cfc4c00c177d94920a0248c41d9b5ac534d69b1aef19f1438"} Nov 04 01:15:16.490156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:16.489866 4312 scope.go:122] "RemoveContainer" containerID="a4a8b8732b481911322921586b6d4ef268fa86f4c499466ab231d6ade59625a1" Nov 04 01:15:16.490307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:16.490291 4312 scope.go:122] "RemoveContainer" containerID="1c375458addc503cfc4c00c177d94920a0248c41d9b5ac534d69b1aef19f1438" Nov 04 01:15:16.490663 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:16.490636 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64_odh-ai-gateway-infra(2561425d-ce5b-4c4a-8d35-08ce31a3742d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" Nov 04 01:15:16.492033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:16.492008 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" event={"ID":"601694d3-057f-4f6a-932a-da07c973acf8","Type":"ContainerStarted","Data":"f84b4abd650a86a3c9270d28ddae0f039e68ab3bde3cfef34a2c22401cc3416f"} Nov 04 01:15:16.520129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:16.520103 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64"] Nov 04 01:15:17.517115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.517082 4312 generic.go:356] "Generic (PLEG): container finished" podID="601694d3-057f-4f6a-932a-da07c973acf8" containerID="f84b4abd650a86a3c9270d28ddae0f039e68ab3bde3cfef34a2c22401cc3416f" exitCode=60 Nov 04 01:15:17.517663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.517637 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" event={"ID":"601694d3-057f-4f6a-932a-da07c973acf8","Type":"ContainerDied","Data":"f84b4abd650a86a3c9270d28ddae0f039e68ab3bde3cfef34a2c22401cc3416f"} Nov 04 01:15:17.517844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.517829 4312 scope.go:122] "RemoveContainer" containerID="2953e4371e5118d59deda85ba1be6cfe00d8083928eaa3215bcb06c49e2b9993" Nov 04 01:15:17.518436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.518418 4312 scope.go:122] "RemoveContainer" containerID="f84b4abd650a86a3c9270d28ddae0f039e68ab3bde3cfef34a2c22401cc3416f" Nov 04 01:15:17.518990 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:17.518964 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv_odh-ai-gateway-infra(601694d3-057f-4f6a-932a-da07c973acf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" podUID="601694d3-057f-4f6a-932a-da07c973acf8" Nov 04 01:15:17.542237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.542207 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv"] Nov 04 01:15:17.688030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.688003 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:17.789621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.789587 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2561425d-ce5b-4c4a-8d35-08ce31a3742d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561425d-ce5b-4c4a-8d35-08ce31a3742d-maas-api-service-ca\") pod \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " Nov 04 01:15:17.789827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.789808 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2561425d-ce5b-4c4a-8d35-08ce31a3742d-kube-api-access-8w8xt\" (UniqueName: \"kubernetes.io/projected/2561425d-ce5b-4c4a-8d35-08ce31a3742d-kube-api-access-8w8xt\") pod \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\" (UID: \"2561425d-ce5b-4c4a-8d35-08ce31a3742d\") " Nov 04 01:15:17.790014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.789994 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2561425d-ce5b-4c4a-8d35-08ce31a3742d-maas-api-service-ca" pod "2561425d-ce5b-4c4a-8d35-08ce31a3742d" (UID: "2561425d-ce5b-4c4a-8d35-08ce31a3742d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:17.790235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.790215 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561425d-ce5b-4c4a-8d35-08ce31a3742d-maas-api-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:17.791923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.791898 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2561425d-ce5b-4c4a-8d35-08ce31a3742d-kube-api-access-8w8xt" pod "2561425d-ce5b-4c4a-8d35-08ce31a3742d" (UID: "2561425d-ce5b-4c4a-8d35-08ce31a3742d"). InnerVolumeSpecName "kube-api-access-8w8xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:17.891689 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:17.891657 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8xt\" (UniqueName: \"kubernetes.io/projected/2561425d-ce5b-4c4a-8d35-08ce31a3742d-kube-api-access-8w8xt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:18.525003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.524919 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" Nov 04 01:15:18.525003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.524935 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64" event={"ID":"2561425d-ce5b-4c4a-8d35-08ce31a3742d","Type":"ContainerDied","Data":"6fe29e5561fbf8c2315ebad452709bd51daa86323e51eded1b6271208a59e0b0"} Nov 04 01:15:18.525574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.525019 4312 scope.go:122] "RemoveContainer" containerID="1c375458addc503cfc4c00c177d94920a0248c41d9b5ac534d69b1aef19f1438" Nov 04 01:15:18.562252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.562218 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64"] Nov 04 01:15:18.570567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.570539 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ba844c-29895915-hcm64"] Nov 04 01:15:18.673050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.673027 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:18.807116 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.807082 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/601694d3-057f-4f6a-932a-da07c973acf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601694d3-057f-4f6a-932a-da07c973acf8-maas-api-service-ca\") pod \"601694d3-057f-4f6a-932a-da07c973acf8\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " Nov 04 01:15:18.807292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.807131 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601694d3-057f-4f6a-932a-da07c973acf8-kube-api-access-clxhx\" (UniqueName: \"kubernetes.io/projected/601694d3-057f-4f6a-932a-da07c973acf8-kube-api-access-clxhx\") pod \"601694d3-057f-4f6a-932a-da07c973acf8\" (UID: \"601694d3-057f-4f6a-932a-da07c973acf8\") " Nov 04 01:15:18.807571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.807547 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601694d3-057f-4f6a-932a-da07c973acf8-maas-api-service-ca" pod "601694d3-057f-4f6a-932a-da07c973acf8" (UID: "601694d3-057f-4f6a-932a-da07c973acf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:18.809428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.809405 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601694d3-057f-4f6a-932a-da07c973acf8-kube-api-access-clxhx" pod "601694d3-057f-4f6a-932a-da07c973acf8" (UID: "601694d3-057f-4f6a-932a-da07c973acf8"). InnerVolumeSpecName "kube-api-access-clxhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:18.908240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.908204 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601694d3-057f-4f6a-932a-da07c973acf8-maas-api-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:18.908240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:18.908234 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clxhx\" (UniqueName: \"kubernetes.io/projected/601694d3-057f-4f6a-932a-da07c973acf8-kube-api-access-clxhx\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:19.534415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:19.534386 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" Nov 04 01:15:19.534868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:19.534388 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv" event={"ID":"601694d3-057f-4f6a-932a-da07c973acf8","Type":"ContainerDied","Data":"d98573a489dd5e4ca978a204005b6ed12d2fea0e529a7915eab5cb7671c20ae1"} Nov 04 01:15:19.534868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:19.534557 4312 scope.go:122] "RemoveContainer" containerID="f84b4abd650a86a3c9270d28ddae0f039e68ab3bde3cfef34a2c22401cc3416f" Nov 04 01:15:19.562168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:19.562140 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv"] Nov 04 01:15:19.566813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:19.566791 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a5115a-29895915-gfmrv"] Nov 04 01:15:20.111919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:20.111885 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" path="/var/lib/kubelet/pods/2561425d-ce5b-4c4a-8d35-08ce31a3742d/volumes" Nov 04 01:15:20.112554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:20.112532 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601694d3-057f-4f6a-932a-da07c973acf8" path="/var/lib/kubelet/pods/601694d3-057f-4f6a-932a-da07c973acf8/volumes" Nov 04 01:15:23.410854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.410820 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6"] Nov 04 01:15:23.411716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411688 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:23.411716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411714 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411729 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411737 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411771 4312 cpu_manager.go:414] "containerMap: removing container" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411778 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411787 4312 cpu_manager.go:414] "containerMap: removing container" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411794 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411824 4312 cpu_manager.go:414] "containerMap: removing container" podUID="c5a33c19-2e92-4037-b663-90074e2ea9f0" containerName="revoke-keys" Nov 04 01:15:23.411917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.411831 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a33c19-2e92-4037-b663-90074e2ea9f0" containerName="revoke-keys" Nov 04 01:15:23.412277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.412000 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:23.412277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.412014 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:23.412277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.412026 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a33c19-2e92-4037-b663-90074e2ea9f0" containerName="revoke-keys" Nov 04 01:15:23.412277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.412036 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:23.412277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.412049 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:23.412277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.412058 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:23.412277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.412068 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:23.417601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.417579 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.423583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.423561 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6"] Nov 04 01:15:23.423904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.423886 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"kube-root-ca.crt\"" Nov 04 01:15:23.425032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.425008 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"openshift-service-ca.crt\"" Nov 04 01:15:23.428297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.428277 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"distinct-w3-a5115a-kserve-self-signed-certs\"" Nov 04 01:15:23.428297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.428283 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"default-dockercfg-458wm\"" Nov 04 01:15:23.611358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.606267 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-tmp-dir\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.611358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.606379 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71320d68-38c3-4423-bf81-0949d491ba50-tls-certs\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.611358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.606448 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccsvd\" (UniqueName: \"kubernetes.io/projected/71320d68-38c3-4423-bf81-0949d491ba50-kube-api-access-ccsvd\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.611358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.606733 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-home\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.611358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.606813 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-model-cache\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.611358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.606881 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-dshm\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.707867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.707790 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71320d68-38c3-4423-bf81-0949d491ba50-tls-certs\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.707867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.707826 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccsvd\" (UniqueName: \"kubernetes.io/projected/71320d68-38c3-4423-bf81-0949d491ba50-kube-api-access-ccsvd\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.708087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.707953 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-home\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.708087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.707992 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-model-cache\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.708087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.708030 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-dshm\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.708087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.708080 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-tmp-dir\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.708443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.708420 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-model-cache\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.708532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.708448 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-tmp-dir\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.708700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.708679 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-home\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.710314 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.710289 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-dshm\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.710428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.710367 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71320d68-38c3-4423-bf81-0949d491ba50-tls-certs\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.719923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.719900 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccsvd\" (UniqueName: \"kubernetes.io/projected/71320d68-38c3-4423-bf81-0949d491ba50-kube-api-access-ccsvd\") pod \"distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:23.735939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:23.735919 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:24.018559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:24.018530 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6"] Nov 04 01:15:24.020021 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:24.019990 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71320d68_38c3_4423_bf81_0949d491ba50.slice/crio-808f716db814cf2866052ba94812cedf6d0746dfa3a2f5eba40050a4535c95dc WatchSource:0}: Error finding container 808f716db814cf2866052ba94812cedf6d0746dfa3a2f5eba40050a4535c95dc: Status 404 returned error can't find the container with id 808f716db814cf2866052ba94812cedf6d0746dfa3a2f5eba40050a4535c95dc Nov 04 01:15:24.589891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:24.589849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" event={"ID":"71320d68-38c3-4423-bf81-0949d491ba50","Type":"ContainerStarted","Data":"d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea"} Nov 04 01:15:24.590284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:24.589902 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" event={"ID":"71320d68-38c3-4423-bf81-0949d491ba50","Type":"ContainerStarted","Data":"808f716db814cf2866052ba94812cedf6d0746dfa3a2f5eba40050a4535c95dc"} Nov 04 01:15:24.631138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:24.631073 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podStartSLOduration=1.6310513599999998 podStartE2EDuration="1.63105136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:24.613224217 +0000 UTC m=+756.636156225" watchObservedRunningTime="2026-11-04 01:15:24.63105136 +0000 UTC m=+756.653983369" Nov 04 01:15:24.739466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:24.739429 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:29.005202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.005115 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g"] Nov 04 01:15:29.005820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.005792 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:29.005820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.005808 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2561425d-ce5b-4c4a-8d35-08ce31a3742d" containerName="cleanup" Nov 04 01:15:29.005947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.005830 4312 cpu_manager.go:414] "containerMap: removing container" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:29.005947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.005836 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601694d3-057f-4f6a-932a-da07c973acf8" containerName="cleanup" Nov 04 01:15:29.009855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.009832 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.021219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.017459 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9902c3\"/\"openshift-service-ca.crt\"" Nov 04 01:15:29.021219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.017581 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9902c3\"/\"distinct-2-w6-9902c3-kserve-self-signed-certs\"" Nov 04 01:15:29.021219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.017657 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9902c3\"/\"default-dockercfg-prz8w\"" Nov 04 01:15:29.021219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.017726 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9902c3\"/\"kube-root-ca.crt\"" Nov 04 01:15:29.021219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.020664 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g"] Nov 04 01:15:29.091865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.091832 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-tmp-dir\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.092027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.091890 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d0569-9744-4daf-aa0e-6b06c891a357-tls-certs\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.092027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.091962 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-model-cache\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.092027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.091983 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-home\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.092027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.092000 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-dshm\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.092183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.092158 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8nw\" (UniqueName: \"kubernetes.io/projected/2d2d0569-9744-4daf-aa0e-6b06c891a357-kube-api-access-bp8nw\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.192998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.192965 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-tmp-dir\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193016 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d0569-9744-4daf-aa0e-6b06c891a357-tls-certs\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193127 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-model-cache\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193162 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-home\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193185 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-dshm\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193293 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8nw\" (UniqueName: \"kubernetes.io/projected/2d2d0569-9744-4daf-aa0e-6b06c891a357-kube-api-access-bp8nw\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193500 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-model-cache\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193524 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-home\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.193657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.193638 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-tmp-dir\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.195304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.195282 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-dshm\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.195421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.195374 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d0569-9744-4daf-aa0e-6b06c891a357-tls-certs\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.205451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.205418 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8nw\" (UniqueName: \"kubernetes.io/projected/2d2d0569-9744-4daf-aa0e-6b06c891a357-kube-api-access-bp8nw\") pod \"distinct-2-w6-9902c3-kserve-56854bcd94-w782g\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.333074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.331672 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:29.475413 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:29.474361 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:15:29.838356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:29.835702 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g"] Nov 04 01:15:29.879356 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:29.875499 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d2d0569_9744_4daf_aa0e_6b06c891a357.slice/crio-2f120f479d94a98dd8ba868d261caea41f146835ff2ca95dacda6acab504c592 WatchSource:0}: Error finding container 2f120f479d94a98dd8ba868d261caea41f146835ff2ca95dacda6acab504c592: Status 404 returned error can't find the container with id 2f120f479d94a98dd8ba868d261caea41f146835ff2ca95dacda6acab504c592 Nov 04 01:15:30.793357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:30.792638 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" event={"ID":"2d2d0569-9744-4daf-aa0e-6b06c891a357","Type":"ContainerStarted","Data":"2f120f479d94a98dd8ba868d261caea41f146835ff2ca95dacda6acab504c592"} Nov 04 01:15:31.064362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:31.059494 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:31.195011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:31.194437 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:31.579415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:31.524430 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:31.712425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:31.711764 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:31.953641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:31.946719 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" event={"ID":"2d2d0569-9744-4daf-aa0e-6b06c891a357","Type":"ContainerStarted","Data":"5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb"} Nov 04 01:15:32.043208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.043141 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podStartSLOduration=4.043121942 podStartE2EDuration="4.043121942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:31.98302176 +0000 UTC m=+764.005953771" watchObservedRunningTime="2026-11-04 01:15:32.043121942 +0000 UTC m=+764.066053950" Nov 04 01:15:32.044319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.044293 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r"] Nov 04 01:15:32.050793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.050772 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.094630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.094555 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r"] Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.142691 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.142755 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.142827 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.142899 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.142938 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.143008 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sx4q\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-kube-api-access-9sx4q\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.143045 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.143087 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71044d9c-6298-48db-b42b-6aab36d15d4d-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.144405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.143143 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.244474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.244390 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sx4q\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-kube-api-access-9sx4q\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.244474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.244442 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.244654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.244481 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71044d9c-6298-48db-b42b-6aab36d15d4d-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.244654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.244539 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.244878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.244859 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.244964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.244955 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.245060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.245051 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.245158 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.245149 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.245223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.245214 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.245415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.245391 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.245637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.245063 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.245852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.245822 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71044d9c-6298-48db-b42b-6aab36d15d4d-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.246098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.246080 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.246212 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.246116 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.248012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.247940 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.248238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.248207 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.257550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.257526 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.257730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.257706 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sx4q\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-kube-api-access-9sx4q\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.332058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.331868 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:32.445096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.445061 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:32.736548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.736525 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r"] Nov 04 01:15:32.737186 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:32.737151 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71044d9c_6298_48db_b42b_6aab36d15d4d.slice/crio-370b0e1a2b3b6e7db12aa0e0f09bd4f5dac3624d025ae3678f005a48f9218520 WatchSource:0}: Error finding container 370b0e1a2b3b6e7db12aa0e0f09bd4f5dac3624d025ae3678f005a48f9218520: Status 404 returned error can't find the container with id 370b0e1a2b3b6e7db12aa0e0f09bd4f5dac3624d025ae3678f005a48f9218520 Nov 04 01:15:32.739698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.739665 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:32.739795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.739731 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:32.739795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.739761 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:32.955814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.955778 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" event={"ID":"71044d9c-6298-48db-b42b-6aab36d15d4d","Type":"ContainerStarted","Data":"e524f96396a9fe01c9002bfd5d5cfd648009ffe3fcddd900ba674ab7beac4686"} Nov 04 01:15:32.956275 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.955840 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" event={"ID":"71044d9c-6298-48db-b42b-6aab36d15d4d","Type":"ContainerStarted","Data":"370b0e1a2b3b6e7db12aa0e0f09bd4f5dac3624d025ae3678f005a48f9218520"} Nov 04 01:15:32.979761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:32.979706 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podStartSLOduration=0.979686606 podStartE2EDuration="979.686606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:32.977456271 +0000 UTC m=+765.000388278" watchObservedRunningTime="2026-11-04 01:15:32.979686606 +0000 UTC m=+765.002618614" Nov 04 01:15:33.445498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:33.445456 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:33.446938 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:33.446911 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:15:33.447098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:33.446963 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 04 01:15:33.737220 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:33.737143 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:33.776060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:33.776026 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:34.033678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:34.033646 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:15:34.445704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:34.445620 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:15:34.445704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:34.445681 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 04 01:15:35.445810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:35.445775 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:15:35.446201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:35.445828 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 04 01:15:36.445655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:36.445620 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:15:36.445800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:36.445669 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 04 01:15:37.447358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:37.446741 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:15:37.447358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:37.446803 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 04 01:15:38.446511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:38.446469 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:15:38.446717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:38.446528 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 04 01:15:39.332545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:39.332502 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:39.345125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:39.345101 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:39.446415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:39.446381 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:15:39.446583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:39.446435 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 04 01:15:40.030501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:40.030474 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:15:40.450278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:40.450200 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:40.450686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:40.450625 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:40.451293 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:40.451276 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:15:42.600667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.600628 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64"] Nov 04 01:15:42.605847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.605830 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.624194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.624168 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"distinct-w0-6dac99-kserve-self-signed-certs\"" Nov 04 01:15:42.629564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.629539 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64"] Nov 04 01:15:42.700889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.700859 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-dshm\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.701058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.701035 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhrf\" (UniqueName: \"kubernetes.io/projected/f879b97c-99be-4398-a09b-866e02a3c8b8-kube-api-access-zvhrf\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.701127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.701103 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-tmp-dir\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.701186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.701144 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-model-cache\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.701225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.701210 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-home\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.701283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.701238 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f879b97c-99be-4398-a09b-866e02a3c8b8-tls-certs\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.804202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.804156 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhrf\" (UniqueName: \"kubernetes.io/projected/f879b97c-99be-4398-a09b-866e02a3c8b8-kube-api-access-zvhrf\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.804924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.804895 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-tmp-dir\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.805572 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.805552 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-model-cache\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.806090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.806070 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-home\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.806544 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.806528 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f879b97c-99be-4398-a09b-866e02a3c8b8-tls-certs\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.807212 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.807192 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-dshm\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.808923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.806467 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-home\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.809069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.805976 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-model-cache\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.809177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.805486 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-tmp-dir\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.821359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.814255 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f879b97c-99be-4398-a09b-866e02a3c8b8-tls-certs\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.822645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.822618 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-dshm\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.829647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.829616 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhrf\" (UniqueName: \"kubernetes.io/projected/f879b97c-99be-4398-a09b-866e02a3c8b8-kube-api-access-zvhrf\") pod \"distinct-w0-6dac99-kserve-7dcb599d8f-jcd64\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:42.919527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:42.919441 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:43.203345 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:43.203302 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64"] Nov 04 01:15:43.204298 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:43.204272 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf879b97c_99be_4398_a09b_866e02a3c8b8.slice/crio-cbaaef04d8ea85029f6934a702cf2677b9d90c43063d5e0803cd99f0f08de4b8 WatchSource:0}: Error finding container cbaaef04d8ea85029f6934a702cf2677b9d90c43063d5e0803cd99f0f08de4b8: Status 404 returned error can't find the container with id cbaaef04d8ea85029f6934a702cf2677b9d90c43063d5e0803cd99f0f08de4b8 Nov 04 01:15:44.052222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:44.052157 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" event={"ID":"f879b97c-99be-4398-a09b-866e02a3c8b8","Type":"ContainerStarted","Data":"a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5"} Nov 04 01:15:44.052222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:44.052211 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" event={"ID":"f879b97c-99be-4398-a09b-866e02a3c8b8","Type":"ContainerStarted","Data":"cbaaef04d8ea85029f6934a702cf2677b9d90c43063d5e0803cd99f0f08de4b8"} Nov 04 01:15:44.074308 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:44.074257 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podStartSLOduration=2.074244266 podStartE2EDuration="2.074244266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:44.07152315 +0000 UTC m=+776.094455182" watchObservedRunningTime="2026-11-04 01:15:44.074244266 +0000 UTC m=+776.097176273" Nov 04 01:15:44.920094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:44.920051 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:47.921361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:47.920567 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984"] Nov 04 01:15:48.055759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.055722 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5"] Nov 04 01:15:48.095472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.083472 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.173189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.161447 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.482775 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.482824 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.482859 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.482883 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9t97\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-kube-api-access-p9t97\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.482913 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33dee613-4f76-4d13-9700-22a0d178c024-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.482936 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.482987 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483013 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b5c39f-d773-4ade-843d-2eeeee33361a-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483046 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483105 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483152 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33dee613-4f76-4d13-9700-22a0d178c024-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483185 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483230 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483259 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bpm\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-kube-api-access-p9bpm\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483328 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483384 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483428 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.483477 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.491926 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:48.503420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.491845 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:48.511357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.509417 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:48.546282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.544776 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:48.546560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.546528 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:48.559633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.553623 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:48.598967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.598924 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.599262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.599225 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.600592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.600536 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.600775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.600759 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.600899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.600883 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9t97\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-kube-api-access-p9t97\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.601017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.601004 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33dee613-4f76-4d13-9700-22a0d178c024-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.601137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.601120 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.601366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.601315 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.601516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.601492 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b5c39f-d773-4ade-843d-2eeeee33361a-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.601676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.601660 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.601859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.601845 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.602031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.601995 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33dee613-4f76-4d13-9700-22a0d178c024-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.602169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.602155 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.602351 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.602311 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.604364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.602486 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bpm\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-kube-api-access-p9bpm\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.604646 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.604611 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.737618 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.737695 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.609016 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.616496 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.631528 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b5c39f-d773-4ade-843d-2eeeee33361a-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.632044 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33dee613-4f76-4d13-9700-22a0d178c024-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.633109 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.653752 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.655442 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.711145 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33dee613-4f76-4d13-9700-22a0d178c024-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.733640 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.734356 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.735324 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984"] Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.738035 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5"] Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.738068 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd"] Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.608615 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.605218 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.767806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.739973 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.801518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.801443 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.854357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.845728 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.854357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.845986 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9t97\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-kube-api-access-p9t97\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.857004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.856971 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:48.929357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.901657 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bpm\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-kube-api-access-p9bpm\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:48.929357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:48.903434 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:49.003055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.003011 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:15:49.006847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.005619 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:49.007489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.007457 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd"] Nov 04 01:15:49.008864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.008731 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177186 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177249 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b81726c-bb5a-4055-b699-50c2042131da-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177313 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177359 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkz6\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-kube-api-access-qhkz6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177418 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177475 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177583 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b81726c-bb5a-4055-b699-50c2042131da-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177623 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.177996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.177713 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279019 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279136 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b81726c-bb5a-4055-b699-50c2042131da-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279176 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279240 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279354 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279394 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b81726c-bb5a-4055-b699-50c2042131da-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279451 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279474 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkz6\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-kube-api-access-qhkz6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.279532 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.282367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.280317 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.299831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.299195 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.299831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.299524 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.300345 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.300304 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.306357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.301123 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b81726c-bb5a-4055-b699-50c2042131da-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.411033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.410638 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b81726c-bb5a-4055-b699-50c2042131da-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.433034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.432962 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.433283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.433114 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.436703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.435807 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkz6\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-kube-api-access-qhkz6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.559688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.559638 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5"] Nov 04 01:15:49.562452 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:49.562329 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dee613_4f76_4d13_9700_22a0d178c024.slice/crio-892b8ed7d475fdc98f68354c7eaa0febf3b24c7fd69b07d9095ecad92ab234b1 WatchSource:0}: Error finding container 892b8ed7d475fdc98f68354c7eaa0febf3b24c7fd69b07d9095ecad92ab234b1: Status 404 returned error can't find the container with id 892b8ed7d475fdc98f68354c7eaa0febf3b24c7fd69b07d9095ecad92ab234b1 Nov 04 01:15:49.565159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.565112 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.565275 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.565209 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.565275 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.565253 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.594024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.593988 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984"] Nov 04 01:15:49.608923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.608884 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.609486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.609457 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.609669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.609649 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.639991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.639962 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:49.988757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.988303 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd"] Nov 04 01:15:49.993604 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:15:49.991559 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b81726c_bb5a_4055_b699_50c2042131da.slice/crio-5777223b477d11d776e871d6e00ab4c908375bed6f073047c82fdf6e0bfe6d84 WatchSource:0}: Error finding container 5777223b477d11d776e871d6e00ab4c908375bed6f073047c82fdf6e0bfe6d84: Status 404 returned error can't find the container with id 5777223b477d11d776e871d6e00ab4c908375bed6f073047c82fdf6e0bfe6d84 Nov 04 01:15:49.995612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.995571 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.995745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.995676 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:49.995745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:49.995727 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:15:50.194371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.193362 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" event={"ID":"36b5c39f-d773-4ade-843d-2eeeee33361a","Type":"ContainerStarted","Data":"2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623"} Nov 04 01:15:50.194371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.193752 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" event={"ID":"36b5c39f-d773-4ade-843d-2eeeee33361a","Type":"ContainerStarted","Data":"d6c6e72d067a2a6334270b65e29f092fbdb686444d5d7d23ec0bfa833dea497a"} Nov 04 01:15:50.195626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.195596 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" event={"ID":"6b81726c-bb5a-4055-b699-50c2042131da","Type":"ContainerStarted","Data":"615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b"} Nov 04 01:15:50.195776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.195663 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" event={"ID":"6b81726c-bb5a-4055-b699-50c2042131da","Type":"ContainerStarted","Data":"5777223b477d11d776e871d6e00ab4c908375bed6f073047c82fdf6e0bfe6d84"} Nov 04 01:15:50.199172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.199146 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" event={"ID":"33dee613-4f76-4d13-9700-22a0d178c024","Type":"ContainerStarted","Data":"9a8b7c1d965b0a15c1f29442e05227586381cecafa1f2aec9d492371b3fcf13f"} Nov 04 01:15:50.199294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.199195 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" event={"ID":"33dee613-4f76-4d13-9700-22a0d178c024","Type":"ContainerStarted","Data":"892b8ed7d475fdc98f68354c7eaa0febf3b24c7fd69b07d9095ecad92ab234b1"} Nov 04 01:15:50.223674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.223605 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podStartSLOduration=3.223585609 podStartE2EDuration="3.223585609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:50.219010127 +0000 UTC m=+782.241942159" watchObservedRunningTime="2026-11-04 01:15:50.223585609 +0000 UTC m=+782.246517615" Nov 04 01:15:50.242561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.242452 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podStartSLOduration=3.242432247 podStartE2EDuration="3.242432247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:50.242383112 +0000 UTC m=+782.265315135" watchObservedRunningTime="2026-11-04 01:15:50.242432247 +0000 UTC m=+782.265364257" Nov 04 01:15:50.272152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.272089 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podStartSLOduration=3.272068149 podStartE2EDuration="3.272068149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:50.266660725 +0000 UTC m=+782.289592750" watchObservedRunningTime="2026-11-04 01:15:50.272068149 +0000 UTC m=+782.295000157" Nov 04 01:15:50.641466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.641425 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:50.917433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.908846 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:50.917974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.917751 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:15:50.918351 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:50.918301 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 04 01:15:51.013918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:51.012661 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:51.013918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:51.012804 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:15:51.013918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:51.012848 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 04 01:15:51.642031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:51.641987 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:15:51.642189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:51.642047 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 04 01:15:51.910416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:51.909897 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:15:51.910416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:51.909998 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 04 01:15:52.018488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:52.018450 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:15:52.018986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:52.018960 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 04 01:15:52.640766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:52.640729 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:15:52.640934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:52.640780 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 04 01:15:52.910052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:52.909958 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:15:52.910492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:52.910450 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 04 01:15:52.925793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:52.925757 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:53.010083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.010042 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:15:53.012317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.012280 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 04 01:15:53.244391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.244286 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:53.335073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.335034 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:15:53.641474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.641439 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:15:53.641656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.641493 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 04 01:15:53.924408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.920746 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:15:53.924408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:53.920814 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 04 01:15:54.009355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:54.009007 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:15:54.009355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:54.009086 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 04 01:15:54.641243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:54.641207 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:15:54.641671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:54.641267 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 04 01:15:54.912926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:54.912830 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:15:54.928588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:54.928547 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 04 01:15:55.022367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:55.022069 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:15:55.022367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:55.022132 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 04 01:15:55.646362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:55.645910 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:15:55.646362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:55.645983 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 04 01:15:55.913786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:55.913690 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:15:55.914033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:55.914005 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 04 01:15:56.012359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:56.006802 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:15:56.012359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:56.006879 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 04 01:15:56.640748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:56.640705 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:15:56.640935 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:56.640794 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 04 01:15:57.348976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.348795 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:57.350924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.349737 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:57.352121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.352100 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:15:57.391036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.391004 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:57.391446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.391424 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:57.392794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.392772 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:15:57.644810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.644705 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:15:57.645196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:57.645164 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 04 01:15:58.645829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:58.645799 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:58.646245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:58.646094 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:58.646794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:15:58.646773 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:15:59.456769 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:15:59.456732 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:16:02.656017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.655977 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz"] Nov 04 01:16:02.830381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.830351 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.910704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.890788 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lph6b\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-kube-api-access-lph6b\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.913083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.913047 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.913402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.913382 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.913550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.913535 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.913645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.913633 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.913792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.913778 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.913957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.913944 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.914066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.914051 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.914304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.914280 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:02.992314 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:02.992273 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz"] Nov 04 01:16:03.011451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.011411 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg"] Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038217 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038314 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph6b\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-kube-api-access-lph6b\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038366 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038467 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038515 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038539 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038595 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038681 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.038710 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.039232 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.040179 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.055553 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg"] Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.055688 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.062856 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.064530 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.065068 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.072917 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.080635 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.114953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.110285 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph6b\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-kube-api-access-lph6b\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.121390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.117895 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.151658 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8860371f-a537-41c7-ae99-7d5c165baed9-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.151719 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8860371f-a537-41c7-ae99-7d5c165baed9-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.151782 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pmw4\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-kube-api-access-5pmw4\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.151895 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.151977 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.152013 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.152036 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.152156 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.171598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.152245 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.258999 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.258958 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.284458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.259656 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.284791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.284764 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.285355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.285235 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.285518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.285313 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.286016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.285996 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.286571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.286551 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.287120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.287098 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.309088 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.309066 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8860371f-a537-41c7-ae99-7d5c165baed9-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.309271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.309252 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8860371f-a537-41c7-ae99-7d5c165baed9-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.310554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.310533 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pmw4\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-kube-api-access-5pmw4\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.332356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.308980 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.334319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.286406 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.334510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.297376 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:03.343360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.310443 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8860371f-a537-41c7-ae99-7d5c165baed9-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.343360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.308262 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.343360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.330856 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.343360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.334250 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8860371f-a537-41c7-ae99-7d5c165baed9-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.344405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.344375 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pmw4\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-kube-api-access-5pmw4\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.501893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.501597 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:03.936362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.935426 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:03.986410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:03.985426 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:04.172360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:04.145415 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:16:04.172360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:04.171613 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:16:04.305165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:04.283425 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:16:04.341362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:04.336243 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:04.742469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:04.742424 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:04.904560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:04.904393 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz"] Nov 04 01:16:05.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:04.954391 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n"] Nov 04 01:16:05.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.022700 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:05.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.022980 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:16:05.083355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.023548 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:05.196009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.161424 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:05.220030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.219894 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:05.401360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.347774 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.403881 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.432270 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.432398 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.432447 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.449625 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.449684 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.449722 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.449758 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.449796 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.450759 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.488184 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-dshm\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.488248 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-home\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.488306 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aac864-b53f-411d-81ef-9e94e015cdad-tls-certs\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.488451 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-tmp-dir\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.488535 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5p2\" (UniqueName: \"kubernetes.io/projected/a7aac864-b53f-411d-81ef-9e94e015cdad-kube-api-access-rx5p2\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.488636 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-model-cache\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.498135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.491660 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"distinct-2-w3-a5115a-kserve-self-signed-certs\"" Nov 04 01:16:05.758580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.745665 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-dshm\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.758580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.745723 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-home\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.758580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.745995 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aac864-b53f-411d-81ef-9e94e015cdad-tls-certs\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.758580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.746167 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-tmp-dir\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.758580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.746227 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5p2\" (UniqueName: \"kubernetes.io/projected/a7aac864-b53f-411d-81ef-9e94e015cdad-kube-api-access-rx5p2\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.758580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.746794 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-model-cache\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.831310 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.828237 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-home\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.915411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.915366 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:05.915839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.915811 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:05.916101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.916077 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:05.920916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.920894 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-tmp-dir\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:05.923735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:05.923715 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n"] Nov 04 01:16:06.001766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.001556 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:16:06.081421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.046889 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:16:06.081421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.047092 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg"] Nov 04 01:16:06.145414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.144512 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:06.161769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.161709 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:06.234133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.180607 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-model-cache\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:06.234133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.180809 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aac864-b53f-411d-81ef-9e94e015cdad-tls-certs\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:06.234133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.190672 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:16:06.235235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.234603 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-dshm\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:06.281866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.281771 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5p2\" (UniqueName: \"kubernetes.io/projected/a7aac864-b53f-411d-81ef-9e94e015cdad-kube-api-access-rx5p2\") pod \"distinct-2-w3-a5115a-kserve-587596f695-tt29n\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:06.347210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.347178 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" event={"ID":"a1355eaa-1dac-49d2-9dab-31ba22187e0e","Type":"ContainerStarted","Data":"63a4a4a9f82e04de0edbbda69dd871e2305ff5b91356f4bc56767d05e98b4532"} Nov 04 01:16:06.347941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.347903 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" event={"ID":"8860371f-a537-41c7-ae99-7d5c165baed9","Type":"ContainerStarted","Data":"01ad24c2b407ed3b09af4794c402ef17e4748c87ee3ea4a702d7eb983261738d"} Nov 04 01:16:06.348201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.348185 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:16:06.348366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.348354 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:16:06.348496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.348484 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:16:06.348834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.348816 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:16:06.350158 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.350138 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:16:06.472839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.472798 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:06.896869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:06.896837 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n"] Nov 04 01:16:06.897638 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:16:06.897610 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7aac864_b53f_411d_81ef_9e94e015cdad.slice/crio-e1f2bb28e0b196c056e0a6eb0dceba8b15fa43d52b2034dc2a5b0b3345847db8 WatchSource:0}: Error finding container e1f2bb28e0b196c056e0a6eb0dceba8b15fa43d52b2034dc2a5b0b3345847db8: Status 404 returned error can't find the container with id e1f2bb28e0b196c056e0a6eb0dceba8b15fa43d52b2034dc2a5b0b3345847db8 Nov 04 01:16:07.406803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.406770 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" event={"ID":"a7aac864-b53f-411d-81ef-9e94e015cdad","Type":"ContainerStarted","Data":"f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953"} Nov 04 01:16:07.407297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.406814 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" event={"ID":"a7aac864-b53f-411d-81ef-9e94e015cdad","Type":"ContainerStarted","Data":"e1f2bb28e0b196c056e0a6eb0dceba8b15fa43d52b2034dc2a5b0b3345847db8"} Nov 04 01:16:07.408822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.408797 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" event={"ID":"a1355eaa-1dac-49d2-9dab-31ba22187e0e","Type":"ContainerStarted","Data":"b2bd8dc05bd8173e5e91e6efbd502af2fcf631be056a9973112bd19f232e6ffa"} Nov 04 01:16:07.410507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.410480 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" event={"ID":"8860371f-a537-41c7-ae99-7d5c165baed9","Type":"ContainerStarted","Data":"265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c"} Nov 04 01:16:07.426033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.425995 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podStartSLOduration=3.425982939 podStartE2EDuration="3.425982939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:07.425912527 +0000 UTC m=+799.448844546" watchObservedRunningTime="2026-11-04 01:16:07.425982939 +0000 UTC m=+799.448914939" Nov 04 01:16:07.482358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.477668 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:07.504358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.503440 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:07.506244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.506215 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:16:07.506408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.506387 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:16:07.530770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.530727 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podStartSLOduration=5.530713977 podStartE2EDuration="5.530713977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:07.506120161 +0000 UTC m=+799.529052184" watchObservedRunningTime="2026-11-04 01:16:07.530713977 +0000 UTC m=+799.553645988" Nov 04 01:16:07.532499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:07.532464 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podStartSLOduration=5.532452955 podStartE2EDuration="5.532452955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:07.528592611 +0000 UTC m=+799.551524635" watchObservedRunningTime="2026-11-04 01:16:07.532452955 +0000 UTC m=+799.555384962" Nov 04 01:16:08.297723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:08.297686 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:08.299032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:08.298999 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:16:08.299260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:08.299232 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 04 01:16:08.502998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:08.502956 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:16:08.503404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:08.503017 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:16:09.297881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:09.297839 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:16:09.298193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:09.298146 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 04 01:16:09.502869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:09.502829 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:16:09.503065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:09.502894 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:16:10.299355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:10.298049 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:16:10.299355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:10.298122 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 04 01:16:10.504163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:10.504126 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:16:10.504619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:10.504177 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:16:11.298359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:11.298086 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:16:11.298359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:11.298165 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 04 01:16:11.503887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:11.503221 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:16:11.503887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:11.503267 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:16:12.321961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:12.311140 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:16:12.321961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:12.311226 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 04 01:16:12.514408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:12.510962 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:16:12.514408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:12.511039 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:16:13.298068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:13.298034 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:16:13.298318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:13.298091 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 04 01:16:13.503232 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:13.503198 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:16:13.503614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:13.503261 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:16:14.302908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:14.302874 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:14.303309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:14.303291 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:14.303949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:14.303929 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:16:14.507111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:14.507081 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:14.507551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:14.507532 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:14.508136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:14.508115 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:16:16.473888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:16.473853 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:16.486065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:16.486040 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:16.560652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:16.560623 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:17.020418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.014143 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd"] Nov 04 01:16:17.053369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.053104 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd"] Nov 04 01:16:17.053835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.053206 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069016 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e128a83-5ec3-4618-b5bf-b32324dc1270-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069103 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069183 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069235 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-kube-api-access-9h92d\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069260 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069390 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069420 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069463 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.069629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.069590 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.171464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.171429 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.171761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.171728 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e128a83-5ec3-4618-b5bf-b32324dc1270-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.171905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.171781 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.171905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.171842 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.171925 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.171951 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-kube-api-access-9h92d\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.171998 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.172084 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.172110 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.172164 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.172324 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.172398 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.172499 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e128a83-5ec3-4618-b5bf-b32324dc1270-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.172599 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.172580 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.174089 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.174062 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.174473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.174453 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.183965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.183939 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.184074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.184032 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-kube-api-access-9h92d\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.371664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.371628 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:17.667976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.667944 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd"] Nov 04 01:16:17.669402 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:16:17.669366 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e128a83_5ec3_4618_b5bf_b32324dc1270.slice/crio-c4db4cb99f0d5262b00157b61eb6b7f377c359f849044423bb9d5448fdab1f05 WatchSource:0}: Error finding container c4db4cb99f0d5262b00157b61eb6b7f377c359f849044423bb9d5448fdab1f05: Status 404 returned error can't find the container with id c4db4cb99f0d5262b00157b61eb6b7f377c359f849044423bb9d5448fdab1f05 Nov 04 01:16:17.672638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.672599 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:17.672724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.672696 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:17.672762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:17.672749 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:18.582887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.582844 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" event={"ID":"0e128a83-5ec3-4618-b5bf-b32324dc1270","Type":"ContainerStarted","Data":"79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc"} Nov 04 01:16:18.583131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.582922 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" event={"ID":"0e128a83-5ec3-4618-b5bf-b32324dc1270","Type":"ContainerStarted","Data":"c4db4cb99f0d5262b00157b61eb6b7f377c359f849044423bb9d5448fdab1f05"} Nov 04 01:16:18.585642 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.585613 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r"] Nov 04 01:16:18.591294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.591266 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.595577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.595524 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"distinct-2-w0-6dac99-kserve-self-signed-certs\"" Nov 04 01:16:18.601863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.601565 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r"] Nov 04 01:16:18.613616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.613558 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podStartSLOduration=2.613540484 podStartE2EDuration="2.613540484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:18.608713387 +0000 UTC m=+810.631645401" watchObservedRunningTime="2026-11-04 01:16:18.613540484 +0000 UTC m=+810.636472493" Nov 04 01:16:18.739512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.739470 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d493695-508b-4253-9229-8465d8ed5626-tls-certs\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.740048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.739547 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-tmp-dir\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.740048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.740002 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-dshm\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.740048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.740037 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8bt\" (UniqueName: \"kubernetes.io/projected/1d493695-508b-4253-9229-8465d8ed5626-kube-api-access-bj8bt\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.740200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.740165 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-home\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.740243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.740201 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-model-cache\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.843604 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-dshm\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.843656 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8bt\" (UniqueName: \"kubernetes.io/projected/1d493695-508b-4253-9229-8465d8ed5626-kube-api-access-bj8bt\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.843742 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-home\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.843772 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-model-cache\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.843823 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d493695-508b-4253-9229-8465d8ed5626-tls-certs\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.843877 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-tmp-dir\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.844545 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-tmp-dir\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.845407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.845195 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-home\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.846117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.846090 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-model-cache\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.877819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.877788 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d493695-508b-4253-9229-8465d8ed5626-tls-certs\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.884355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.882054 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8bt\" (UniqueName: \"kubernetes.io/projected/1d493695-508b-4253-9229-8465d8ed5626-kube-api-access-bj8bt\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.918325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.905695 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-dshm\") pod \"distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:18.918325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:18.912246 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:19.373415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.372794 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:19.376501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.376469 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:16:19.376915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.376757 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:16:19.378727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.378701 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r"] Nov 04 01:16:19.380219 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:16:19.380192 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d493695_508b_4253_9229_8465d8ed5626.slice/crio-443b58831420cd2adfba7f0c7a025e9acf0615eeb413829df40f900750dd0479 WatchSource:0}: Error finding container 443b58831420cd2adfba7f0c7a025e9acf0615eeb413829df40f900750dd0479: Status 404 returned error can't find the container with id 443b58831420cd2adfba7f0c7a025e9acf0615eeb413829df40f900750dd0479 Nov 04 01:16:19.591650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.591609 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" event={"ID":"1d493695-508b-4253-9229-8465d8ed5626","Type":"ContainerStarted","Data":"8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733"} Nov 04 01:16:19.591650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.591660 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" event={"ID":"1d493695-508b-4253-9229-8465d8ed5626","Type":"ContainerStarted","Data":"443b58831420cd2adfba7f0c7a025e9acf0615eeb413829df40f900750dd0479"} Nov 04 01:16:19.612476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.612426 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podStartSLOduration=1.6124127129999999 podStartE2EDuration="1.612412713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:19.609640699 +0000 UTC m=+811.632572719" watchObservedRunningTime="2026-11-04 01:16:19.612412713 +0000 UTC m=+811.635344719" Nov 04 01:16:19.915956 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:19.915924 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:20.373867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:20.373809 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:16:20.374093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:20.373883 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:16:21.372717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:21.372678 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:16:21.373222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:21.372754 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:16:22.374357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:22.373884 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:16:22.374357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:22.373989 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:16:23.372771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:23.372727 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:16:23.372960 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:23.372800 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:16:24.372580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:24.372550 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:16:24.372970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:24.372598 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:16:25.376078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:25.376048 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:25.376478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:25.376437 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:25.377183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:25.377161 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:16:28.913639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:28.913603 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:28.926223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:28.926198 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:29.578811 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:16:29.578774 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:16:30.727359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:30.705465 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:30.818358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:30.818278 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:16:31.028417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.027484 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.271358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.258421 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.326416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.318474 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.357465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.357418 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.551366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.550466 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.673415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.662412 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.751547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.749420 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.829854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.828084 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:31.829854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.828487 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:16:31.829854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:31.829121 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:16:32.043422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.042662 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:32.043422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.043398 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:16:32.259356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.259083 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:16:32.486410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.477355 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:32.515359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.502565 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:16:32.616365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.572906 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:16:32.702201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.632181 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:16:32.852715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:32.852671 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:16:33.000622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.000575 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:16:33.052852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.052804 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:16:33.095391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.095349 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:16:33.260821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.260724 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:33.549075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.547892 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:33.549075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.548793 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:16:33.710377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.708452 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:16:33.754543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.750428 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 04 01:16:33.795253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.788909 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:33.899488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.895229 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:33.977674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:33.977628 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:34.017358 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:16:33.997300 4312 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.89s" Nov 04 01:16:34.043816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.043788 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h"] Nov 04 01:16:34.076360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.057979 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:16:34.120360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.110354 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:34.269520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.269415 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:16:34.542141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.542098 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:34.609357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.557278 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:34.637082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.572448 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:16:34.638244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.617560 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:16:34.638520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.638504 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:16:34.639434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.617901 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:34.742358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.739467 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:34.781532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.771649 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"distinct-2-w2-cffa80-kserve-self-signed-certs\"" Nov 04 01:16:34.802746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.802643 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:16:34.828761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.823291 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-dshm\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:34.828761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.823384 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6lz\" (UniqueName: \"kubernetes.io/projected/6a4c5c00-4791-4127-815b-b8715decf72f-kube-api-access-qt6lz\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:34.828761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.823618 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-home\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:34.828761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.823703 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-tmp-dir\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:34.828761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.823772 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c5c00-4791-4127-815b-b8715decf72f-tls-certs\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:34.828761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.823828 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-model-cache\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:34.934360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.927218 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:16:34.934360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.927435 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h"] Nov 04 01:16:34.934360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.927495 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:34.934360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:34.927516 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm"] Nov 04 01:16:35.011478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.011427 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:35.011899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.011864 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:16:35.017362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.013739 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-home\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.017362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.013847 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-tmp-dir\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.017362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.013897 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c5c00-4791-4127-815b-b8715decf72f-tls-certs\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.017362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.013951 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-model-cache\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.017362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.014016 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-dshm\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.017362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.014066 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6lz\" (UniqueName: \"kubernetes.io/projected/6a4c5c00-4791-4127-815b-b8715decf72f-kube-api-access-qt6lz\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.017362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.014997 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-home\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.043697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.043663 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm"] Nov 04 01:16:35.043854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.043721 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf"] Nov 04 01:16:35.137408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.105246 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-tmp-dir\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.179597 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.139244 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-model-cache\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.179808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.159655 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:35.180243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.159772 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.181739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.167214 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-dshm\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.196361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.133175 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:16:35.221364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.207919 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9"] Nov 04 01:16:35.277143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.277105 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c5c00-4791-4127-815b-b8715decf72f-tls-certs\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.290887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.288466 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.296788 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.296750 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:16:35.459423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.403712 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6lz\" (UniqueName: \"kubernetes.io/projected/6a4c5c00-4791-4127-815b-b8715decf72f-kube-api-access-qt6lz\") pod \"distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.493117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.492782 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:35.557799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.557753 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:35.563172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.563142 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf"] Nov 04 01:16:35.622494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.619196 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:16:35.622494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.619276 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:16:35.622494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.619302 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9"] Nov 04 01:16:35.653691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.653659 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:16:35.669530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.669501 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.675896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.675864 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:16:35.677645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.677619 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.677845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.677826 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.678206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.678186 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.678385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.678369 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.678517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.678502 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.678623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.678610 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.678751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.678737 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.678855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.678841 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.678979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.678963 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.679112 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.679094 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.679273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.679257 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.679553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.679533 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.679766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.679747 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.679909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.679894 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqpj\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-kube-api-access-jhqpj\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.680030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680017 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.680141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680127 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.680283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680267 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.680430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680413 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.680558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680541 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.680710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680694 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.680883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680869 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.681013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.680994 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.681190 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.681173 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrvx\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-kube-api-access-hrrvx\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.681327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.681302 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.681651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.681630 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttch\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-kube-api-access-jttch\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.721094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.721003 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.743574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.743450 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.787649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.770478 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:16:35.815954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.789756 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:16:35.871388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.867048 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.871748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.871724 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.871946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.871917 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.872138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.872124 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.872276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.872260 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.872426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.872406 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrvx\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-kube-api-access-hrrvx\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.872594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.872557 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876294 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttch\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-kube-api-access-jttch\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876424 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876470 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876506 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876539 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876642 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876695 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876736 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876788 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876870 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876905 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.876981 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877049 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877132 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877183 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877228 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877271 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqpj\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-kube-api-access-jhqpj\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877322 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877371 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.877440 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.878414 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.882350 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.882661 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.882966 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.883224 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.883501 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.883840 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.884090 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.886180 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.886572 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.890384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.886852 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.892034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.891421 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.892034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.891729 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.893376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.892312 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.893376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.892806 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.899764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.894529 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:16:35.917406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.917082 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.920152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.919977 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:35.939934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.921169 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.940441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.935294 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:16:35.943097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.943080 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:16:35.969266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.961302 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:35.973301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.973234 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:16:35.984475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.982703 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:35.984475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.983156 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:35.984690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:35.984582 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:16:36.011032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.010581 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:16:36.019897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.019866 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:36.021389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.021357 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:36.021669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.021645 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttch\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-kube-api-access-jttch\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:36.032726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.022537 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrvx\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-kube-api-access-hrrvx\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:36.032726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.022322 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:36.032726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.024084 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqpj\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-kube-api-access-jhqpj\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:36.032726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.024206 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:36.173720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.173688 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:36.192693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.192665 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:36.294245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.290836 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h"] Nov 04 01:16:36.305209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.305183 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:36.484895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.484853 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.485010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.484924 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.485010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.484950 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.492078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.492050 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" event={"ID":"6a4c5c00-4791-4127-815b-b8715decf72f","Type":"ContainerStarted","Data":"c8dacd6457ad4901a5131150cdc48b4a17ee0cacb4ef9628892892cee9a9d5ff"} Nov 04 01:16:36.492279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.492238 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf"] Nov 04 01:16:36.492471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.492454 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" event={"ID":"6a4c5c00-4791-4127-815b-b8715decf72f","Type":"ContainerStarted","Data":"d4e416455396bd0794864ddb355e790248e80c460e8b13887d675ff81ec969fb"} Nov 04 01:16:36.496083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.496051 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:36.515506 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:16:36.515467 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4b7300_8fc6_4173_b3d0_f1a65d296fbd.slice/crio-26589589d08290743bc6cb6943eb59e3017240334673cafb9744a6e0b781e0aa WatchSource:0}: Error finding container 26589589d08290743bc6cb6943eb59e3017240334673cafb9744a6e0b781e0aa: Status 404 returned error can't find the container with id 26589589d08290743bc6cb6943eb59e3017240334673cafb9744a6e0b781e0aa Nov 04 01:16:36.515795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.515711 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm"] Nov 04 01:16:36.517601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.517551 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podStartSLOduration=5.517534152 podStartE2EDuration="5.517534152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:36.515010567 +0000 UTC m=+828.537942573" watchObservedRunningTime="2026-11-04 01:16:36.517534152 +0000 UTC m=+828.540466160" Nov 04 01:16:36.518378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.518346 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.518447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.518414 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.518486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.518448 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.621518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.621074 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9"] Nov 04 01:16:36.625559 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:16:36.625529 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc098cc9b_21ac_4ece_a7a7_1ec5d6bf2c55.slice/crio-5df9d6eca478d38dcea2cea8fdd680f365f1a487dc7531cf934c62abd1b99498 WatchSource:0}: Error finding container 5df9d6eca478d38dcea2cea8fdd680f365f1a487dc7531cf934c62abd1b99498: Status 404 returned error can't find the container with id 5df9d6eca478d38dcea2cea8fdd680f365f1a487dc7531cf934c62abd1b99498 Nov 04 01:16:36.631149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.628816 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.631149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.628915 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:36.631149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:36.628969 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:16:37.509835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.509788 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" event={"ID":"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55","Type":"ContainerStarted","Data":"6771ed881ea0585229b0e34a110ac47302f6233bbc1ebbde58f83926a45eb197"} Nov 04 01:16:37.510310 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.509855 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" event={"ID":"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55","Type":"ContainerStarted","Data":"5df9d6eca478d38dcea2cea8fdd680f365f1a487dc7531cf934c62abd1b99498"} Nov 04 01:16:37.512445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.512417 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" event={"ID":"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd","Type":"ContainerStarted","Data":"1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54"} Nov 04 01:16:37.512639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.512624 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" event={"ID":"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd","Type":"ContainerStarted","Data":"26589589d08290743bc6cb6943eb59e3017240334673cafb9744a6e0b781e0aa"} Nov 04 01:16:37.515672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.515629 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" event={"ID":"82a77e5e-42bb-4b2e-a233-942284a5e1e9","Type":"ContainerStarted","Data":"a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955"} Nov 04 01:16:37.515672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.515675 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" event={"ID":"82a77e5e-42bb-4b2e-a233-942284a5e1e9","Type":"ContainerStarted","Data":"766a53b2e5b3a467b2ee66739d10414a3fc7a29fdf0549b0fa6bb17124f9f1eb"} Nov 04 01:16:37.540769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.540675 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podStartSLOduration=5.540653967 podStartE2EDuration="5.540653967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:37.533322407 +0000 UTC m=+829.556254414" watchObservedRunningTime="2026-11-04 01:16:37.540653967 +0000 UTC m=+829.563585966" Nov 04 01:16:37.621364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.619038 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podStartSLOduration=5.619018113 podStartE2EDuration="5.619018113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:37.580660964 +0000 UTC m=+829.603592973" watchObservedRunningTime="2026-11-04 01:16:37.619018113 +0000 UTC m=+829.641950121" Nov 04 01:16:37.632240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:37.631893 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podStartSLOduration=5.631873091 podStartE2EDuration="5.631873091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:37.615023547 +0000 UTC m=+829.637955558" watchObservedRunningTime="2026-11-04 01:16:37.631873091 +0000 UTC m=+829.654805102" Nov 04 01:16:38.224357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.223503 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:16:38.224357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.223601 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:16:38.224357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.223907 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:16:38.224357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.223943 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:16:38.304454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.284899 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:38.304775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.304749 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:38.314553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.314502 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:38.323501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.323466 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:16:38.323731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:38.323705 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:16:39.181567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:39.181530 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:16:39.182147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:39.182115 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:16:39.211581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:39.211529 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:16:39.211870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:39.211832 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:16:39.318360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:39.310322 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:16:39.318360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:39.310405 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:16:40.175358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:40.175260 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:16:40.175358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:40.175343 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:16:40.196260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:40.196226 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:16:40.196872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:40.196839 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:16:40.325495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:40.320572 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:16:40.325495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:40.320634 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:16:41.208365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:41.194661 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:16:41.208365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:41.194721 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:16:41.208365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:41.195128 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:16:41.208365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:41.195161 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:16:41.320358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:41.309899 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:16:41.320358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:41.309959 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:16:42.188358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:42.174086 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:16:42.188358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:42.174173 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:16:42.221359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:42.199439 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:16:42.221359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:42.199527 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:16:42.306357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:42.305885 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:16:42.306357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:42.305992 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:16:43.207376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:43.194799 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:16:43.207376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:43.194876 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:16:43.220816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:43.220680 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:16:43.220816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:43.220749 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:16:43.312083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:43.312044 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:16:43.323106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:43.323058 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:16:44.175201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:44.175168 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:16:44.175608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:44.175560 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:16:44.193628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:44.193563 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:16:44.193802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:44.193627 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:16:44.305574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:44.305544 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:16:44.306867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:44.305611 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:16:45.341134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.340583 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:45.342582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.341942 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:45.342975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.342921 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:16:45.360820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.360792 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:45.361253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.361233 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:45.363291 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.363265 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:16:45.498563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.498530 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:45.513159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.513132 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:45.517133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.517111 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:45.517509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.517488 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:45.518415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.518397 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:16:45.667307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:16:45.667225 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:16:59.459079 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:16:59.458990 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:17:01.940359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:01.932484 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z"] Nov 04 01:17:02.203463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.154746 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:02.203463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.157647 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:02.346359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.343988 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:02.548418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.497153 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:02.661365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.631326 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw"] Nov 04 01:17:02.717265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.663462 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:02.717265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.664098 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:02.755792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.754871 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z"] Nov 04 01:17:02.786402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.759540 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:02.786402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.761437 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:02.815435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.788323 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:02.819358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.815773 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:02.823358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.819645 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtc7\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-kube-api-access-hwtc7\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:02.827356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.823587 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:02.840360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.831588 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:02.864360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.840709 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4847a0b-f8fa-46e8-873c-76e99b10588c-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953069 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953200 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953266 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953309 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshvt\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-kube-api-access-rshvt\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953353 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953395 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953428 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f2bc6-f790-4c9b-af7a-c183240e3b68-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953470 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953513 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953567 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953617 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.953665 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.037498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:02.962321 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:03.038673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.037664 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw"] Nov 04 01:17:03.067478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.067129 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb"] Nov 04 01:17:03.074361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.068445 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.084520 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.086408 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088076 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088165 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088224 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtc7\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-kube-api-access-hwtc7\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088265 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088385 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088431 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4847a0b-f8fa-46e8-873c-76e99b10588c-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088464 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088571 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088630 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088673 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshvt\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-kube-api-access-rshvt\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088699 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088748 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088788 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f2bc6-f790-4c9b-af7a-c183240e3b68-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088833 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088880 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088934 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.088980 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.104858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.089026 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.120891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.120857 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4847a0b-f8fa-46e8-873c-76e99b10588c-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.121588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.121562 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.122272 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.122250 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.138355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.122822 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.138355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.123137 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.138355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.127650 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.158370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.154246 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.159710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.158839 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.159710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.159280 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f2bc6-f790-4c9b-af7a-c183240e3b68-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.159710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.159377 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.159974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.159953 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.162409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.160235 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.208113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.169616 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.218364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.208985 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.218364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.209502 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:03.218364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.213868 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb"] Nov 04 01:17:03.225523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.221877 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:17:03.258371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.209750 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.258371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.246406 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f"] Nov 04 01:17:03.259107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.259075 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.370480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.370394 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.385399 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.385532 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.385624 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwmd\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-kube-api-access-mfwmd\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.385670 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.385718 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.385796 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.385991 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.386023 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.391365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.386066 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.529854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.529817 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.642146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.642051 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.649868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.646015 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.650271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.650244 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.650470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.650449 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.650673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.650655 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.650916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.650896 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.651121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.651104 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwmd\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-kube-api-access-mfwmd\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.651274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.651252 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.651463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.651443 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.652069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.652045 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.652202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.648392 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.659558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.656468 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.659558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.656848 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.659558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.657041 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:17:03.659558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.657069 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f"] Nov 04 01:17:03.659558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.658015 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:03.660230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.660206 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.670977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.664322 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.670977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.665033 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.670977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.665617 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.707305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.697458 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:03.707305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.698876 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:17:03.707305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.700694 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.707305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.701168 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:17:03.714414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.710285 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.729427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.728663 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:03.741359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.741301 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshvt\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-kube-api-access-rshvt\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:03.759529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.759492 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtc7\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-kube-api-access-hwtc7\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.761464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.761410 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:03.774364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.774303 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:03.774976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.774948 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:17:03.799706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.799644 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:03.849194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:03.837472 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwmd\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-kube-api-access-mfwmd\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:04.009555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.009453 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.010277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.010250 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:17:04.018326 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.018076 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:04.046511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.046231 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:04.113508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.113473 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.113853 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.113824 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfds\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-kube-api-access-ctfds\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.114048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.114026 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.114261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.114244 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.114403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.114386 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.114627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.114608 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.306634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.289497 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.306634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.289764 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.306634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.289845 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.306634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.268792 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:04.306634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.116560 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.307193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.152131 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:04.334074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.152092 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:04.402360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.280888 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404050 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404228 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404296 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404381 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404427 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfds\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-kube-api-access-ctfds\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404470 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404546 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404575 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.416377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.404656 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.485196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.441876 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.621366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.603946 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.690360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.620904 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.690360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.623525 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.690360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.635528 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:04.690360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.649188 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:17:04.690360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.649366 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:17:04.690360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.649427 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:17:04.690360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.686620 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.812358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.701837 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.812358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.702591 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.812358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.770519 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.812358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.771324 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.848312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.848212 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:04.879081 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:04.879045 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfds\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-kube-api-access-ctfds\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:05.100419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.080603 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:05.122405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.107028 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:05.132358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.128963 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:05.343308 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.314051 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:05.371894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.371783 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:05.567059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.567011 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:05.590323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.590280 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:05.765009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.764903 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:05.767200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.767165 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:05.769773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.769745 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:05.779829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.779779 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:05.860367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.835146 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:05.863362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.863297 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:05.955395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:05.955325 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:06.500357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.431848 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:06.558718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.402258 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:06.559110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.489358 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:06.559366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.494893 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:06.584927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.571160 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:06.608785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.608544 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:06.668346 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.668300 4312 trace.go:236] Trace[1770990502]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-56hwl" (04-Nov-2026 01:17:05.417) (total time: 1250ms): Nov 04 01:17:06.668346 ip-10-0-136-96 kubenswrapper[4312]: Trace[1770990502]: [1.250689537s] [1.250689537s] END Nov 04 01:17:06.920610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.920560 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:06.922251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.922192 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 04 01:17:06.922509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.920643 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:06.957362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.948868 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:07.196496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:06.920593 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:07.227554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.132085 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:07.675666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.675598 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:07.702099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.702050 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 04 01:17:07.911360 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:17:07.907725 4312 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.419s" Nov 04 01:17:08.431361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.951729 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" podUID="ba71124b-dc83-408b-8e5c-1b9abd6fac9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.480361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.998440 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.480361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.998481 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.480361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.998550 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.480361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.998571 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:08.480361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.998591 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.480361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:07.998614 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:08.499356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:08.192434 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.499356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:08.224990 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:08.499356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:08.234693 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.716394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:08.716278 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:08.841291 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:08.827911 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:09.343304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.311658 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded" Nov 04 01:17:09.576952 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.576910 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:09.600620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.600510 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:09.603010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.602954 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:09.633360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.604827 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:09.663304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.663253 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:09.782974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.782927 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:09.787360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.784196 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:10.024383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.842067 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:10.024383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:09.611709 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:10.582470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:10.582418 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:10.583174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:10.583142 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:10.692412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:10.662163 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:10.693873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:10.693828 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:10.694206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:10.694167 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:10.779666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:10.779600 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 04 01:17:11.062858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.047797 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:11.062858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.048223 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:11.062858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.048551 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:11.102359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.101553 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:11.138531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.138483 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:11.201115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.144555 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:11.201115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.197486 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:11.224973 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.224914 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:11.228922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.228556 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.338719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.326930 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.338719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.327120 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.338719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.327236 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.338719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.333841 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.338719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.335612 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.584549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.583098 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:11.652116 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.652069 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:11.687819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.687772 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:11.761889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.760478 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.761889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.760601 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.761889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.760659 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821194 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw"] Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821255 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z"] Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821322 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821373 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821390 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb"] Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821437 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821468 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821498 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821528 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821557 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821586 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821633 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:17:11.825568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.821657 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f"] Nov 04 01:17:11.829611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.829586 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" event={"ID":"b4847a0b-f8fa-46e8-873c-76e99b10588c","Type":"ContainerStarted","Data":"dd677a415ea73be3e9b74f0341154af9ee10ab98cc81c8af7a1f5db2c00f7425"} Nov 04 01:17:11.831700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.831676 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" event={"ID":"c5b6ab23-7925-4dc8-a773-f70db1a2fe06","Type":"ContainerStarted","Data":"32555fcf8c718d6c1798acc6b0e08f03a06ec3f6b4b1ba4446a137ed27f10048"} Nov 04 01:17:11.833485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.833458 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" event={"ID":"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857","Type":"ContainerStarted","Data":"65d537aa4f7b0da726cf2bf55f803a824251d97256db464780aeecc6e54b3b5e"} Nov 04 01:17:11.835114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:11.835089 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" event={"ID":"551f2bc6-f790-4c9b-af7a-c183240e3b68","Type":"ContainerStarted","Data":"f2be21a782fbca09586beb8bc0d864271f3fc32546b69ea1cc712973cd0e150f"} Nov 04 01:17:12.895357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:12.895073 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" event={"ID":"b4847a0b-f8fa-46e8-873c-76e99b10588c","Type":"ContainerStarted","Data":"05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6"} Nov 04 01:17:12.908766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:12.908728 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" event={"ID":"c5b6ab23-7925-4dc8-a773-f70db1a2fe06","Type":"ContainerStarted","Data":"17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e"} Nov 04 01:17:12.916878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:12.916845 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" event={"ID":"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857","Type":"ContainerStarted","Data":"8cd64a5f27260f300ff6a4454101d8a920120d7429ea5df4766d6bca392d3bcb"} Nov 04 01:17:12.917152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:12.917134 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" event={"ID":"551f2bc6-f790-4c9b-af7a-c183240e3b68","Type":"ContainerStarted","Data":"eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95"} Nov 04 01:17:12.967770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:12.967705 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podStartSLOduration=11.96768606 podStartE2EDuration="11.96768606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:12.956747528 +0000 UTC m=+864.979679564" watchObservedRunningTime="2026-11-04 01:17:12.96768606 +0000 UTC m=+864.990618068" Nov 04 01:17:13.020979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.020946 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:13.021291 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.021268 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:13.021498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.021470 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:13.058113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.058075 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:13.058320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.058205 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:13.058320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.058249 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:13.060722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.060668 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podStartSLOduration=12.060649862 podStartE2EDuration="12.060649862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:12.985835397 +0000 UTC m=+865.008767404" watchObservedRunningTime="2026-11-04 01:17:13.060649862 +0000 UTC m=+865.083581869" Nov 04 01:17:13.073454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.073397 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podStartSLOduration=11.073380211 podStartE2EDuration="11.073380211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:13.042253513 +0000 UTC m=+865.065185611" watchObservedRunningTime="2026-11-04 01:17:13.073380211 +0000 UTC m=+865.096312219" Nov 04 01:17:13.144408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.133462 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podStartSLOduration=11.133439542 podStartE2EDuration="11.133439542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:13.094859311 +0000 UTC m=+865.117791348" watchObservedRunningTime="2026-11-04 01:17:13.133439542 +0000 UTC m=+865.156371550" Nov 04 01:17:13.294221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.294166 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:13.295360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.295312 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:13.295602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.295574 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:13.469361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.465595 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:13.469361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.467015 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:13.469361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:13.467064 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:14.023420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.023383 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:14.024023 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.023992 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:14.052358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.048642 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:14.052358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.048700 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:14.336359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.335465 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:14.336359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.335551 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:14.460477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.459947 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:14.460477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.460013 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:14.485816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.485764 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:14.532359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.503855 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:14.651949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.651837 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:14.652284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.652259 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:14.654118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.654087 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:14.760446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.740586 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:14.804364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.796911 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:14.804364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.799138 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:14.804364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.801889 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:14.859360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:14.858647 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.024131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.024035 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:15.039355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.030263 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:15.049012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.048984 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:15.049299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.049270 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:15.284366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.283543 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:15.284366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.283605 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:15.348392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.341431 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.496360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.488517 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:15.496360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.489294 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:17:15.532355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.512967 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.693416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.550812 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:15.693416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.692815 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:15.693416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.693243 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:17:15.749661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.737621 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.749661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.744927 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:17:15.749661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.746218 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.815361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.782424 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:15.815361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.813925 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.848360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.822549 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.848360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.822822 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.848360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.823372 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:17:15.848360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.823453 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:17:15.848360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.823489 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:17:15.876361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.850865 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:15.876361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.851161 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:17:15.882557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.880725 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:15.882557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:15.881033 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:17:16.029976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.029946 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:16.030538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.030506 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:16.223951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.223915 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:16.224227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.224196 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:16.327243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.323515 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:16.327243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.323593 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:16.386634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.386587 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:16.546153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.511425 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:16.576468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.519673 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:16.580520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.580406 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:16.608560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.549524 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:16.734522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.708430 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:16.739477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.739437 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:16.748633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.748589 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:16.798358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.788457 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:17:16.837906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.837813 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:16.839415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.839381 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:16.840401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:16.839440 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:17.046375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.033513 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:17.072412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.057696 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:17.159423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.073947 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:17.159423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.074006 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:17.159423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.074093 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:17.159423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.074117 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:17.300772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.300735 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:17.301006 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.300979 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:17.491414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.466743 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:17.491832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.491806 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:17.492018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.491718 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.567359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.555568 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg"] Nov 04 01:17:17.589379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.567690 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.589379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.572589 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.589379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.575998 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.687189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.687150 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg"] Nov 04 01:17:17.715110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.713815 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.743780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.743676 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.763958 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.763920 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.764229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.764203 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.764459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.764436 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.764704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.764680 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.764829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.764813 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.764966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.764950 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836f5631-21d0-4b71-99cd-1d0a82c8a855-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.765079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.765065 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.765181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.765165 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.765404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.765386 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wzr\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-kube-api-access-j8wzr\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:17.812242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.812200 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.850576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.850532 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.851295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.851252 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:17.937789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.937727 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:18.006418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.967294 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.007470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.007424 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.007608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.007521 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836f5631-21d0-4b71-99cd-1d0a82c8a855-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.007608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.007570 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.007731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.007608 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.007796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.007764 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wzr\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-kube-api-access-j8wzr\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.008037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.008017 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.008111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.008066 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.008185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.008168 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.014365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.009184 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.014365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.009777 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.046385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.043646 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:18.144358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.059880 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.144358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:17.978842 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.144358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.060666 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836f5631-21d0-4b71-99cd-1d0a82c8a855-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.144358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.086714 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:18.144358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.093025 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:18.144358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.093090 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:18.144358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.141939 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wzr\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-kube-api-access-j8wzr\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.183052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.183016 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:18.194390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.183316 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:18.264414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.254928 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.454365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.315867 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:18.454365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.315946 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:18.454365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.350199 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.539359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.536442 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.589360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.554256 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:18.589360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.554368 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:17:18.589360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.577160 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:18.589360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.586759 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:18.589360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.586817 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:18.589360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.587083 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:18.714510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:18.714463 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:19.055462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.052048 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:19.058390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.058347 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:19.312822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.059045 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:19.335293 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.059120 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:19.336556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.059148 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:19.337499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.090903 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:19.337899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.091684 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:19.344140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.344118 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:19.383376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.282795 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded" Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.285265 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.336745 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.241294 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.412584 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.241855 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.414432 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.282930 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:19.417380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.414502 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:19.484473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.482749 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:19.484473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.482821 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:19.582979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.579362 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:19.768072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:19.752232 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:20.002516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.002435 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:20.106371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.084956 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:20.106371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.084997 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:20.106371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.085022 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:20.106371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.085044 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:20.106371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.093551 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:17:20.397356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.396814 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:20.407373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.400131 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 04 01:17:20.427372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.425722 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:20.543574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.543530 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:20.544399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.543647 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:20.601356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.601290 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:20.650807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.643068 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:20.673356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.651037 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:20.682492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.666973 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:20.734732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.734699 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:20.835458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.828137 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:20.884718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.882514 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:20.884718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.882587 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:20.924058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.923967 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg"] Nov 04 01:17:20.964354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.946925 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:20.981101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.980833 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:17:20.990617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.990581 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:17:20.997589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:20.797291 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:21.016520 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:17:21.010381 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836f5631_21d0_4b71_99cd_1d0a82c8a855.slice/crio-6e45367f049073e0944918a995d64399b48fc0a86731e013668c9879ef13fa72\": RecentStats: unable to find data in memory cache]" Nov 04 01:17:21.016520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.011824 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:17:21.020428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.020391 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:21.020697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.020670 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:21.020862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.020844 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:21.031587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.031559 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:21.035762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.035732 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:21.067366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.066008 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:17:21.070907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.070867 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" event={"ID":"836f5631-21d0-4b71-99cd-1d0a82c8a855","Type":"ContainerStarted","Data":"6e45367f049073e0944918a995d64399b48fc0a86731e013668c9879ef13fa72"} Nov 04 01:17:21.080125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.079843 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:17:21.124798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.102908 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:17:21.124798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.102966 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 04 01:17:21.125362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.125295 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:17:21.277034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.276034 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:17:21.282753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.282639 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:17:21.282753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.282699 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:17:21.476416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.440505 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:21.512640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.477160 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:21.534835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:21.534790 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:22.018830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.018793 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:17:22.019158 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.019121 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:17:22.207357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.193311 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podStartSLOduration=6.193289038 podStartE2EDuration="6.193289038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:22.181082063 +0000 UTC m=+874.204014071" watchObservedRunningTime="2026-11-04 01:17:22.193289038 +0000 UTC m=+874.216221046" Nov 04 01:17:22.237380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.224400 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" event={"ID":"836f5631-21d0-4b71-99cd-1d0a82c8a855","Type":"ContainerStarted","Data":"7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a"} Nov 04 01:17:22.517368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.514515 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:17:22.517368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.514586 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 01:17:22.604619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.604499 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:22.606605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.606543 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:17:22.606856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.606802 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:17:22.821387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.815658 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:22.841773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:22.841738 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:17:23.088671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.088579 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:23.153927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.153899 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:17:23.601781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.601743 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:17:23.602411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.602381 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:17:23.687359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.686803 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:23.692892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.692866 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:17:23.783355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.775305 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:23.793299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:23.792744 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:17:24.600635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:24.600601 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:17:24.600798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:24.600657 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:17:25.600401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:25.600318 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:17:25.600912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:25.600442 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:17:26.600730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:26.600671 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:17:26.601253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:26.600756 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:17:27.601322 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:27.601278 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:17:27.602063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:27.601947 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:17:28.601261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:28.601224 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:17:28.601515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:28.601283 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:17:29.532395 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:17:29.532290 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:17:29.766490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:29.766458 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:29.766968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:29.766914 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:29.767918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:29.767888 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:17:32.573278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:32.573243 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v"] Nov 04 01:17:32.578512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:32.578478 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:32.598109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:32.598067 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v"] Nov 04 01:17:32.929514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:32.929234 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk"] Nov 04 01:17:32.975011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:32.974985 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:32.995250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:32.974782 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk"] Nov 04 01:17:32.998422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:32.998384 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.004494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.004464 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.004733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.004716 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46565c-06b6-43ea-8d37-da745354cad9-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.004939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.004926 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.005074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.005062 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.005259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.005246 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.005419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.005406 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4c5\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-kube-api-access-rj4c5\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.005669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.005652 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46565c-06b6-43ea-8d37-da745354cad9-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.005932 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.005917 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.122625 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46565c-06b6-43ea-8d37-da745354cad9-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.122736 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.122772 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.122801 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.122853 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.122883 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.122953 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.123026 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.123066 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.123126 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46565c-06b6-43ea-8d37-da745354cad9-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.123162 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.131713 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46565c-06b6-43ea-8d37-da745354cad9-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.132095 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.132384 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.179378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.132683 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.211410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.201774 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2kc7\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-kube-api-access-m2kc7\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.211410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.201888 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.211410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.201917 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.211410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.202003 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.211410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.202036 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4c5\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-kube-api-access-rj4c5\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.211410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.202109 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.211410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.202177 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.231454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.217742 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.258746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.258700 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46565c-06b6-43ea-8d37-da745354cad9-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.305914 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.305977 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.306051 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.306130 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.306212 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.306309 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.306354 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2kc7\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-kube-api-access-m2kc7\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.306488 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.316357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.306556 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.363356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.359610 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.363356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.362618 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.363356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.362952 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.363677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.363547 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.365254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.365213 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.514400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.368419 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.514400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.385738 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.514400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.512104 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.514400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.514301 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2kc7\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-kube-api-access-m2kc7\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.550403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.533620 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.550403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.535034 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4c5\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-kube-api-access-rj4c5\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.551168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.551104 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:33.734037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.733993 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:33.743419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:33.735998 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:34.569540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:34.567451 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:34.600663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:34.599455 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:34.643477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:34.642458 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:34.731783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:34.731724 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:34.785811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:34.785467 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:34.851441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:34.842142 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:34.852965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:34.852423 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:35.079160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.079119 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:35.412557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.346435 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:35.576360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.572662 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:35.576360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.573397 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:17:35.617370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.612510 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:35.617370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.613287 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:17:35.669420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.651966 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:35.669420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.652413 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:17:35.735612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.735200 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:35.736469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.736190 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:17:35.920415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.918842 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:35.991390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.964424 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:35.991390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.965253 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:17:35.991390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:35.965315 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:17:36.015389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.005832 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:36.059410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.042946 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:36.089437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.086726 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:36.089437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.087145 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:17:36.453499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.348392 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:36.527356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.526838 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:36.822639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.680177 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:36.825902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.684407 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:36.827570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.684639 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:36.832725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.749726 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:36.934836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.934797 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:37.012627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:36.974495 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:37.021540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.008900 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:37.083531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.073724 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:37.150390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.137841 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:37.201449 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:17:37.181136 4312 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.011s" Nov 04 01:17:37.248422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.214543 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:17:37.254677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.254637 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:37.254923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.254910 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:17:37.404072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.378471 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:37.431566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.412405 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:37.431566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.412503 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:37.431566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.412552 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:37.889404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.882530 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:37.889404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.882634 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:37.889404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.882680 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:17:37.918624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.896565 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:37.918624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.915237 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:37.918624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.916181 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:37.923356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.920024 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:38.014532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.014489 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:38.050469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.020752 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:38.050469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.022281 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:38.088672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:37.519323 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 04 01:17:38.204675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.131436 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:38.205150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.205023 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:38.819424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.737897 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:38.931902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.897553 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:38.946519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:38.937673 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:39.074482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.074364 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:39.145776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.145723 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:39.175548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.150191 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:39.177792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.177740 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:39.267356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.263640 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:39.296070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.296019 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:39.430450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.421730 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:39.482428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.480764 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Nov 04 01:17:39.482428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.481599 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded" Nov 04 01:17:39.565468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.556806 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:39.565468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.556883 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:39.619490 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:17:39.601209 4312 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.353s" Nov 04 01:17:39.752525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.749194 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:39.765395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.762101 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:39.765395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.762182 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:39.947062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.946999 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:39.966779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:39.956398 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v"] Nov 04 01:17:40.020105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.020004 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.137745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.137701 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.159366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.156425 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.188030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.180494 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.244776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.244558 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.268781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.268420 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.328375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.310112 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.361673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.361630 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 04 01:17:40.440404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.428361 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.448644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.440685 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.540901 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.540438 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:40.540901 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.540550 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.610614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.604438 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:40.610614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.604512 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.611015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.610977 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" event={"ID":"51f907ae-3f89-40f0-b5be-96fba4eff5c1","Type":"ContainerStarted","Data":"c38f2ca7874aee052bf82e3388364ecc6c2e9fe52504d920cef896d42cbb92c1"} Nov 04 01:17:40.611185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.611167 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" event={"ID":"1b46565c-06b6-43ea-8d37-da745354cad9","Type":"ContainerStarted","Data":"cf3a590c0c7a33443c3d3c45ee9ba4ab90cbecd42504fa01baab9ed9fec81690"} Nov 04 01:17:40.611322 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.611309 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk"] Nov 04 01:17:40.766551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.765459 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:40.948416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:40.947942 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:41.022721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.021484 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.163359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.161904 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.163359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.162262 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.186957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.184468 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.259436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.255859 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:41.266473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.266437 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:41.266993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.266700 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.275358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.272426 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.334816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.327001 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:41.400357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.381462 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" event={"ID":"51f907ae-3f89-40f0-b5be-96fba4eff5c1","Type":"ContainerStarted","Data":"2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b"} Nov 04 01:17:41.411352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.403213 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" event={"ID":"1b46565c-06b6-43ea-8d37-da745354cad9","Type":"ContainerStarted","Data":"5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f"} Nov 04 01:17:41.436605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.436539 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podStartSLOduration=9.436518002 podStartE2EDuration="9.436518002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:41.419901057 +0000 UTC m=+893.442833088" watchObservedRunningTime="2026-11-04 01:17:41.436518002 +0000 UTC m=+893.459450011" Nov 04 01:17:41.480462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.480418 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:41.484292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.481184 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:41.484557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.484526 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.518609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.518430 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:41.518609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.518491 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.654692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.654636 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.809408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.809373 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:41.826233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.826186 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:41.850202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.826083 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:41.850948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.809733 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:41.850948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.850821 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:41.850948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.849514 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:41.850948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.850881 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:41.932375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.891512 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:41.932375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:41.891575 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:42.046552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.030957 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:42.046552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.031952 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:42.217531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.217422 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:42.218464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.218429 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:42.245644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.245602 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:42.289850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.269348 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:42.289850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.284559 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:42.336182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.336141 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:42.521420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.518416 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:42.746956 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.746916 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:42.747246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.747211 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:42.751024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.748607 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:42.751229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.751203 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:42.852968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.851434 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:42.987372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.983498 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:42.987372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:42.983568 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.278411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.264758 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.291487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.291436 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.318356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.309445 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.318356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.310779 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:43.318356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.311055 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:43.318356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.316213 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:43.343616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.343567 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.359451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.347304 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.493779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.493714 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podStartSLOduration=11.49369358 podStartE2EDuration="11.49369358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:41.515626418 +0000 UTC m=+893.538558426" watchObservedRunningTime="2026-11-04 01:17:43.49369358 +0000 UTC m=+895.516625588" Nov 04 01:17:43.495073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.495030 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:43.495199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.495101 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.508358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.502410 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr"] Nov 04 01:17:43.517701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.517633 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:43.518031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.517972 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:43.521604 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.521571 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 04 01:17:43.529565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.529441 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:43.604412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.577246 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr"] Nov 04 01:17:43.630518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.577357 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:43.713577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.678599 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:17:43.713577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.679287 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:17:43.821417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.766817 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:43.875760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.875709 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:43.876018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.852483 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:43.876443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.767041 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:43.876628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.876605 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:43.880919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.880888 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:43.881097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.881077 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh68w\" (UniqueName: \"kubernetes.io/projected/a697aba5-b71b-499a-b06f-43f635f8b862-kube-api-access-bh68w\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:43.904358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:43.884981 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:44.087424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.054792 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:44.458586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.458488 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:44.525389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.464811 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh68w\" (UniqueName: \"kubernetes.io/projected/a697aba5-b71b-499a-b06f-43f635f8b862-kube-api-access-bh68w\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:44.650777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.327667 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:44.676459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.327728 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:44.712500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.334423 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:44.712731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.334478 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 04 01:17:44.714263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.347448 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:17:44.714473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.319311 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:44.715820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.347801 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:44.719097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.484889 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:44.722352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.273472 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:44.754200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.297480 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:44.778385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.603811 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:44.778912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.778866 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:44.882524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.882478 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:44.914751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:44.914718 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:45.108472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.084851 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:45.108472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.084920 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.108472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.085255 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:45.108472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.085294 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:45.108472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.086711 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh68w\" (UniqueName: \"kubernetes.io/projected/a697aba5-b71b-499a-b06f-43f635f8b862-kube-api-access-bh68w\") pod \"maas-api-revoke-sub-534009bf-xkfkr\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:45.139403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.113432 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:45.197609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.114443 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" podUID="a7f1bc94-14c3-4783-9aad-2efc85a264e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 04 01:17:45.197609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.121111 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" podUID="870c4be8-527c-43ad-a63f-39dd02718c55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.197609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.121154 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" podUID="61ca9501-204d-4daf-a56b-0794b0f8f6d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Nov 04 01:17:45.197609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.149668 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:45.287359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.286570 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:45.376422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.371431 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:17:45.376422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.371500 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.686424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.677523 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.712292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.712252 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:17:45.714310 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.714276 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.717182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.717157 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:17:45.718580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.718554 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:45.727446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.727407 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:45.731457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.731424 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.744265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.744220 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:45.744526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.744500 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:45.745645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.745624 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:45.746086 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.746065 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:45.757556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.757313 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:45.781969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.781890 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.903482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.903435 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:45.929760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.922528 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:17:45.948558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:45.944798 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:17:46.550621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.550575 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 04 01:17:46.814937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.778490 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:46.906604 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.906557 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:46.935784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.907203 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:46.936797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.907497 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:17:46.940104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.940078 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:17:46.940318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.940303 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:17:46.943214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.943188 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr"] Nov 04 01:17:46.944830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:46.944805 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:47.003229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.003189 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:47.003467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.003258 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:47.013504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.013452 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:47.051381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.044252 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:47.051381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.044301 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:47.097415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.076987 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:47.098193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.098164 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:17:47.120717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.108471 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:17:47.229510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.229485 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:17:47.232365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.231608 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:47.238977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.238942 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:17:47.300692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.300659 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:17:47.763356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.744922 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:47.763356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.745002 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:47.763356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.745517 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:47.763356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:47.745559 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:48.005062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.004987 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" event={"ID":"a697aba5-b71b-499a-b06f-43f635f8b862","Type":"ContainerStarted","Data":"6f05d1c5bf6ce7c1d150e9ed57fdfbe7d68c7bf61b03a77b9f596b308db55d4a"} Nov 04 01:17:48.005062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.005037 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" event={"ID":"a697aba5-b71b-499a-b06f-43f635f8b862","Type":"ContainerStarted","Data":"9c23bc3adae29c82dceb0f499d4e3819edd782f73ddd25b0a257806ab48816c0"} Nov 04 01:17:48.026048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.025998 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" podStartSLOduration=5.025983514 podStartE2EDuration="5.025983514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:48.023798913 +0000 UTC m=+900.046730914" watchObservedRunningTime="2026-11-04 01:17:48.025983514 +0000 UTC m=+900.048915520" Nov 04 01:17:48.735129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.735094 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:48.741767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.741735 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:48.741902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.741781 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:48.743904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.743883 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:48.744130 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.744099 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:48.744219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:48.744140 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:49.090357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:49.089085 4312 generic.go:356] "Generic (PLEG): container finished" podID="a697aba5-b71b-499a-b06f-43f635f8b862" containerID="6f05d1c5bf6ce7c1d150e9ed57fdfbe7d68c7bf61b03a77b9f596b308db55d4a" exitCode=0 Nov 04 01:17:49.090357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:49.089249 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" event={"ID":"a697aba5-b71b-499a-b06f-43f635f8b862","Type":"ContainerDied","Data":"6f05d1c5bf6ce7c1d150e9ed57fdfbe7d68c7bf61b03a77b9f596b308db55d4a"} Nov 04 01:17:49.750358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:49.743793 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:49.750358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:49.743871 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:49.774354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:49.764368 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:49.774354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:49.764607 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:50.512791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.512764 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:50.697369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.697257 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a697aba5-b71b-499a-b06f-43f635f8b862-kube-api-access-bh68w\" (UniqueName: \"kubernetes.io/projected/a697aba5-b71b-499a-b06f-43f635f8b862-kube-api-access-bh68w\") pod \"a697aba5-b71b-499a-b06f-43f635f8b862\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " Nov 04 01:17:50.704354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.697717 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-openshift-ca\") pod \"a697aba5-b71b-499a-b06f-43f635f8b862\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " Nov 04 01:17:50.704620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.704592 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-xks-ca\") pod \"a697aba5-b71b-499a-b06f-43f635f8b862\" (UID: \"a697aba5-b71b-499a-b06f-43f635f8b862\") " Nov 04 01:17:50.735545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.735251 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-openshift-ca" pod "a697aba5-b71b-499a-b06f-43f635f8b862" (UID: "a697aba5-b71b-499a-b06f-43f635f8b862"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.742925 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.742982 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.743838 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.743878 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.803500 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-xks-ca" pod "a697aba5-b71b-499a-b06f-43f635f8b862" (UID: "a697aba5-b71b-499a-b06f-43f635f8b862"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.806495 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a697aba5-b71b-499a-b06f-43f635f8b862-kube-api-access-bh68w" pod "a697aba5-b71b-499a-b06f-43f635f8b862" (UID: "a697aba5-b71b-499a-b06f-43f635f8b862"). InnerVolumeSpecName "kube-api-access-bh68w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.807887 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh68w\" (UniqueName: \"kubernetes.io/projected/a697aba5-b71b-499a-b06f-43f635f8b862-kube-api-access-bh68w\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.807915 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:50.827621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:50.807931 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a697aba5-b71b-499a-b06f-43f635f8b862-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:51.121720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:51.121685 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" Nov 04 01:17:51.121866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:51.121716 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr" event={"ID":"a697aba5-b71b-499a-b06f-43f635f8b862","Type":"ContainerDied","Data":"9c23bc3adae29c82dceb0f499d4e3819edd782f73ddd25b0a257806ab48816c0"} Nov 04 01:17:51.121866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:51.121769 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c23bc3adae29c82dceb0f499d4e3819edd782f73ddd25b0a257806ab48816c0" Nov 04 01:17:51.742664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:51.742623 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:17:51.743073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:51.742695 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:17:51.744001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:51.743977 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:17:51.744091 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:51.744025 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:17:52.751924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:52.751895 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:52.753204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:52.753183 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:17:52.775432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:52.775393 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:52.776863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:52.776841 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:17:57.021695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.021640 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:57.186677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.186637 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:57.187540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.187508 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:57.353541 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.353481 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:57.419298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.419242 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:17:57.475674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.475618 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:57.843382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.841399 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:17:57.854700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.854660 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:57.855257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.854930 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:57.860503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:57.860473 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:17:58.046494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.046443 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:58.047527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.047503 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:17:58.281507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.265515 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:58.430358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.334171 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:58.457776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.385209 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:58.484480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.484435 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:58.513649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.508859 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:58.737872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.737765 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:58.873363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.844587 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.873669 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.875428 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.876308 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.037313 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.037407 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.037475 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.037581 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.037641 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.037695 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:17:59.044362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:58.901630 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:59.152991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.137153 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:17:59.197389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.197325 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:17:59.376629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.375899 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded" Nov 04 01:17:59.376629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.376023 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:17:59.544533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.520850 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:17:59.722810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:17:59.722710 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:00.071475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.046519 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:00.071475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.047392 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:18:00.071475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.047660 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:00.071475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.049682 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Nov 04 01:18:00.071475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.050035 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:18:00.126363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.085404 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:00.126363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.085778 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:00.126363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.107147 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:00.126363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.107224 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:00.201867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.170408 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:00.212922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.212885 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:00.439218 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.433496 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:00.605359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.604448 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:00.736710 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:18:00.736619 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:18:00.790359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.765109 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 04 01:18:00.958153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:00.775108 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:01.056037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.055972 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.057631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.057589 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.067740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.067700 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.091493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.091445 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.118191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.091736 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.120513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.099764 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:01.120760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.120719 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.171671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.171618 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.329809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.329708 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.456360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.452572 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:01.482995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.482960 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:01.483252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.483222 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.558966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.558922 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:01.578473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.578414 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.579014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.577473 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:01.579201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.579176 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.604937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.604821 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.674374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.669420 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:01.965443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:01.963557 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.056811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.056769 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.062002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.061947 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.096782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.096465 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.131579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.131525 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:02.133800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.133755 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:02.173798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.173754 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.400358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.399438 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.485532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.485486 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.609237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.608586 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:02.976356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:02.972654 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:03.061466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.061408 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:18:03.063030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.063003 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:18:03.128862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.128814 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:03.135456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.135417 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:03.135988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.135960 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:03.176119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.176065 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:03.409360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.408445 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:03.489491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.489441 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:03.520494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.520446 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:03.619557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.619504 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:03.938514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.938464 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:03.993567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:03.973419 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:04.064849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.064804 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.065436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.065385 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:18:04.159124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.159081 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:04.161759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.161724 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.186557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.165690 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.187263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.176960 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.420509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.420458 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.510784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.510737 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.631477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.631429 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.772421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.771426 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:04.975356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:04.974427 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:05.072513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.072467 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:05.102173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.102132 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:18:05.254384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.217452 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:05.254384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.217845 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:05.254384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.218134 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:05.379439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.350576 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:05.458360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.437804 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:05.577157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.553241 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:05.751707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.657376 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:05.816368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:05.812642 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:18:06.243258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.243202 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:06.243679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.243560 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:06.243838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.243792 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:06.321111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.321047 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:06.388133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.343177 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:18:06.404856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.404798 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:06.472868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.472827 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:06.498442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.474143 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:06.722838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.722638 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:18:06.723032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:06.723002 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 04 01:18:07.114171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.114141 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:18:07.265819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.251528 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:07.265819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.251925 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:07.265819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.252208 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:18:07.423438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.388673 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:07.423438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.389060 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:07.480357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.475711 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:07.480357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.476721 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:07.480357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.480189 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:07.502525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.502482 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:07.558385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.558353 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f"] Nov 04 01:18:07.609352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.605491 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a697aba5-b71b-499a-b06f-43f635f8b862" containerName="revoke-keys" Nov 04 01:18:07.609352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.605525 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a697aba5-b71b-499a-b06f-43f635f8b862" containerName="revoke-keys" Nov 04 01:18:07.723059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.722964 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:07.729082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.727188 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a697aba5-b71b-499a-b06f-43f635f8b862" containerName="revoke-keys" Nov 04 01:18:07.775432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.775405 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:18:07.775666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.775651 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f"] Nov 04 01:18:07.775824 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.775813 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:18:07.775973 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.775960 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:18:07.777124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.777107 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.864835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.864802 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8f95141e-openshift-default-dockercfg-ts4t9\"" Nov 04 01:18:07.960359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960057 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-envoy\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960145 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-credential-socket\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960190 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-data\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960276 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-podinfo\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960307 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-socket\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960388 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmwh\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-kube-api-access-mvmwh\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960583 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-token\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960643 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0139988f-7aa7-4de8-8209-a0a5cda050e2-istiod-ca-cert\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:07.960864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:07.960709 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-certs\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.067463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.065026 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-token\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.067463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.065103 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0139988f-7aa7-4de8-8209-a0a5cda050e2-istiod-ca-cert\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.067463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.065166 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-certs\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.067463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.065379 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-envoy\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.117104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.117065 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0139988f-7aa7-4de8-8209-a0a5cda050e2-istiod-ca-cert\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.117438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.117405 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:08.187276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.187242 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-credential-socket\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.187594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.187567 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-data\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.187802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.187785 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-podinfo\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.187937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.187920 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-socket\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.188082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.188065 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmwh\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-kube-api-access-mvmwh\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.194762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.189196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-certs\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.228377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.196354 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-socket\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.303815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.303768 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:08.305056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.305022 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-token\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.308043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.308018 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmwh\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-kube-api-access-mvmwh\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.308252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.308237 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:18:08.311225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.311208 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:18:08.312394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.312374 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-credential-socket\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.314906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.314885 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-envoy\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.316969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.316947 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-data\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.317578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.317530 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-podinfo\") pod \"e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.357070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.357030 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:18:08.357246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.357169 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:18:08.357246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.357208 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:18:08.473221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.473156 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:08.811360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.810889 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f"] Nov 04 01:18:08.822372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.822320 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:18:08.822475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.822422 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:18:08.822475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:08.822464 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:18:09.271847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.271765 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" event={"ID":"0139988f-7aa7-4de8-8209-a0a5cda050e2","Type":"ContainerStarted","Data":"9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891"} Nov 04 01:18:09.271847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.271836 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" event={"ID":"0139988f-7aa7-4de8-8209-a0a5cda050e2","Type":"ContainerStarted","Data":"cb1e23fdbe84404f8f177839a668cf695a03ab32f2f9fbc155754d5d74411e5d"} Nov 04 01:18:09.297723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.297675 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" podStartSLOduration=2.297659934 podStartE2EDuration="2.297659934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:09.295449838 +0000 UTC m=+921.318381881" watchObservedRunningTime="2026-11-04 01:18:09.297659934 +0000 UTC m=+921.320591941" Nov 04 01:18:09.319586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.319558 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8"] Nov 04 01:18:09.338680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.338011 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:09.341400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.341362 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8"] Nov 04 01:18:09.346068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.346044 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9cc22d46\"" Nov 04 01:18:09.356506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.356475 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2"] Nov 04 01:18:09.362248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.362189 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.368592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.366845 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9cc22d46-dockercfg-tcqls\"" Nov 04 01:18:09.372967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.372945 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2"] Nov 04 01:18:09.468404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.465686 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9x85\" (UniqueName: \"kubernetes.io/projected/204d1390-12f5-45e8-a024-c55df88b9605-kube-api-access-k9x85\") pod \"payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:09.468404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.465785 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204d1390-12f5-45e8-a024-c55df88b9605-plugins-config-volume\") pod \"payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:09.468404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.465859 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617f8d0e-7252-4d6c-859b-74168c0ca47f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.468404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.466025 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjw7\" (UniqueName: \"kubernetes.io/projected/617f8d0e-7252-4d6c-859b-74168c0ca47f-kube-api-access-jnjw7\") pod \"payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.504359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.504312 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:09.583356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.570585 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjw7\" (UniqueName: \"kubernetes.io/projected/617f8d0e-7252-4d6c-859b-74168c0ca47f-kube-api-access-jnjw7\") pod \"payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.583974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.583938 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9x85\" (UniqueName: \"kubernetes.io/projected/204d1390-12f5-45e8-a024-c55df88b9605-kube-api-access-k9x85\") pod \"payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:09.584539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.584510 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204d1390-12f5-45e8-a024-c55df88b9605-plugins-config-volume\") pod \"payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:09.607933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.607892 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617f8d0e-7252-4d6c-859b-74168c0ca47f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.609226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.609196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617f8d0e-7252-4d6c-859b-74168c0ca47f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.609415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.585581 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204d1390-12f5-45e8-a024-c55df88b9605-plugins-config-volume\") pod \"payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:09.658356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.640960 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9x85\" (UniqueName: \"kubernetes.io/projected/204d1390-12f5-45e8-a024-c55df88b9605-kube-api-access-k9x85\") pod \"payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:09.675781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.675696 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjw7\" (UniqueName: \"kubernetes.io/projected/617f8d0e-7252-4d6c-859b-74168c0ca47f-kube-api-access-jnjw7\") pod \"payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.681742 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.681667 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:09.845207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.841689 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:09.959425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:09.959391 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:10.111738 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:18:10.111480 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617f8d0e_7252_4d6c_859b_74168c0ca47f.slice/crio-a0ffb502c5ad512e8e768764e4206ad5cb0c6b0f37b57026ca74b910b2d4578a WatchSource:0}: Error finding container a0ffb502c5ad512e8e768764e4206ad5cb0c6b0f37b57026ca74b910b2d4578a: Status 404 returned error can't find the container with id a0ffb502c5ad512e8e768764e4206ad5cb0c6b0f37b57026ca74b910b2d4578a Nov 04 01:18:10.125084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.125054 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2"] Nov 04 01:18:10.248204 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:18:10.248176 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204d1390_12f5_45e8_a024_c55df88b9605.slice/crio-3da9a4f544607d0e8be34163e341dac94a755b1c93fb9191b4b73800643a2b74 WatchSource:0}: Error finding container 3da9a4f544607d0e8be34163e341dac94a755b1c93fb9191b4b73800643a2b74: Status 404 returned error can't find the container with id 3da9a4f544607d0e8be34163e341dac94a755b1c93fb9191b4b73800643a2b74 Nov 04 01:18:10.250253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.250224 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8"] Nov 04 01:18:10.281672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.281640 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" event={"ID":"617f8d0e-7252-4d6c-859b-74168c0ca47f","Type":"ContainerStarted","Data":"0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143"} Nov 04 01:18:10.281822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.281711 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:10.281822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.281736 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" event={"ID":"617f8d0e-7252-4d6c-859b-74168c0ca47f","Type":"ContainerStarted","Data":"a0ffb502c5ad512e8e768764e4206ad5cb0c6b0f37b57026ca74b910b2d4578a"} Nov 04 01:18:10.283543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.283516 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" event={"ID":"204d1390-12f5-45e8-a024-c55df88b9605","Type":"ContainerStarted","Data":"3da9a4f544607d0e8be34163e341dac94a755b1c93fb9191b4b73800643a2b74"} Nov 04 01:18:10.283826 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.283802 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:10.284793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.284774 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:18:10.300798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:10.300752 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" podStartSLOduration=1.300738415 podStartE2EDuration="1.300738415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:10.299446971 +0000 UTC m=+922.322378999" watchObservedRunningTime="2026-11-04 01:18:10.300738415 +0000 UTC m=+922.323670421" Nov 04 01:18:11.303058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:11.303014 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" event={"ID":"204d1390-12f5-45e8-a024-c55df88b9605","Type":"ContainerStarted","Data":"b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf"} Nov 04 01:18:11.303682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:11.303655 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:11.327458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:11.327398 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" podStartSLOduration=2.327380897 podStartE2EDuration="2.327380897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:11.32596101 +0000 UTC m=+923.348893018" watchObservedRunningTime="2026-11-04 01:18:11.327380897 +0000 UTC m=+923.350312903" Nov 04 01:18:12.274464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.274425 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s"] Nov 04 01:18:12.280892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.280863 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:12.294207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.294181 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s"] Nov 04 01:18:12.313281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.313238 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db301d6-995a-45a2-86ff-1fb9de40e228-plugins-config-volume\") pod \"payload-processing-e2e-ait-383a864b-745f85645b-rxp4s\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:12.313776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.313356 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jw2s\" (UniqueName: \"kubernetes.io/projected/6db301d6-995a-45a2-86ff-1fb9de40e228-kube-api-access-2jw2s\") pod \"payload-processing-e2e-ait-383a864b-745f85645b-rxp4s\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:12.369171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.369121 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-383a864b\"" Nov 04 01:18:12.369605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.369583 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-383a864b-dockercfg-q49x9\"" Nov 04 01:18:12.424418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.415683 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db301d6-995a-45a2-86ff-1fb9de40e228-plugins-config-volume\") pod \"payload-processing-e2e-ait-383a864b-745f85645b-rxp4s\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:12.424418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.415771 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jw2s\" (UniqueName: \"kubernetes.io/projected/6db301d6-995a-45a2-86ff-1fb9de40e228-kube-api-access-2jw2s\") pod \"payload-processing-e2e-ait-383a864b-745f85645b-rxp4s\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:12.424418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.417176 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db301d6-995a-45a2-86ff-1fb9de40e228-plugins-config-volume\") pod \"payload-processing-e2e-ait-383a864b-745f85645b-rxp4s\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:12.722378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.722326 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jw2s\" (UniqueName: \"kubernetes.io/projected/6db301d6-995a-45a2-86ff-1fb9de40e228-kube-api-access-2jw2s\") pod \"payload-processing-e2e-ait-383a864b-745f85645b-rxp4s\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:12.914038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:12.912897 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:13.282952 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:18:13.282914 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db301d6_995a_45a2_86ff_1fb9de40e228.slice/crio-ef99c16ca54ed6d1d3f8d9b9497573b3e1568c05fc82f20f5b0620dad38aaefe WatchSource:0}: Error finding container ef99c16ca54ed6d1d3f8d9b9497573b3e1568c05fc82f20f5b0620dad38aaefe: Status 404 returned error can't find the container with id ef99c16ca54ed6d1d3f8d9b9497573b3e1568c05fc82f20f5b0620dad38aaefe Nov 04 01:18:13.284550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:13.284517 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s"] Nov 04 01:18:13.544355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:13.541849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" event={"ID":"6db301d6-995a-45a2-86ff-1fb9de40e228","Type":"ContainerStarted","Data":"ef99c16ca54ed6d1d3f8d9b9497573b3e1568c05fc82f20f5b0620dad38aaefe"} Nov 04 01:18:14.550717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:14.550681 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" event={"ID":"6db301d6-995a-45a2-86ff-1fb9de40e228","Type":"ContainerStarted","Data":"9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758"} Nov 04 01:18:14.551126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:14.550767 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:14.568713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:14.568651 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" podStartSLOduration=2.568637634 podStartE2EDuration="2.568637634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:14.568560195 +0000 UTC m=+926.591492205" watchObservedRunningTime="2026-11-04 01:18:14.568637634 +0000 UTC m=+926.591569641" Nov 04 01:18:19.703686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.703652 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7"] Nov 04 01:18:19.709856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.709832 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.716985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.716940 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-383a864b-serving-cert\"" Nov 04 01:18:19.721194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.721170 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7"] Nov 04 01:18:19.721438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.721411 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-metrics-certs\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.721601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.721574 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ce0cb4-d037-465d-b09a-1e6b32d198e1-openshift-service-ca\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.721729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.721716 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-maas-api-tls\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.721845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.721826 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5z27\" (UniqueName: \"kubernetes.io/projected/84ce0cb4-d037-465d-b09a-1e6b32d198e1-kube-api-access-w5z27\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.823698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.823666 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-metrics-certs\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.823892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.823797 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ce0cb4-d037-465d-b09a-1e6b32d198e1-openshift-service-ca\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.823970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.823901 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-maas-api-tls\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.824029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.823974 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5z27\" (UniqueName: \"kubernetes.io/projected/84ce0cb4-d037-465d-b09a-1e6b32d198e1-kube-api-access-w5z27\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.824577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.824551 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ce0cb4-d037-465d-b09a-1e6b32d198e1-openshift-service-ca\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.826230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.826208 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-metrics-certs\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.826354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.826250 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-maas-api-tls\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:19.836440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:19.836411 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5z27\" (UniqueName: \"kubernetes.io/projected/84ce0cb4-d037-465d-b09a-1e6b32d198e1-kube-api-access-w5z27\") pod \"maas-api-e2e-ait-383a864b-fb47bb76-hjcl7\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:20.040771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:20.033865 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:20.564854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:20.564827 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7"] Nov 04 01:18:20.655090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:20.655057 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" event={"ID":"84ce0cb4-d037-465d-b09a-1e6b32d198e1","Type":"ContainerStarted","Data":"95a13b7d96390ca44ba964cb774011d4f044a6dcb992d51744d36e88d448342d"} Nov 04 01:18:21.306679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:21.306650 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:21.722358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:21.714463 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" event={"ID":"84ce0cb4-d037-465d-b09a-1e6b32d198e1","Type":"ContainerStarted","Data":"4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a"} Nov 04 01:18:21.722358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:21.715416 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:21.763359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:21.762385 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" podStartSLOduration=2.177828402 podStartE2EDuration="2.762362245s" totalImagesPullingTime="584.533843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:21.7470683 +0000 UTC m=+933.770000308" watchObservedRunningTime="2026-11-04 01:18:21.762362245 +0000 UTC m=+933.785294253" Nov 04 01:18:22.477355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:22.470578 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:25.563229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:25.563190 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:18:28.735071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:28.734996 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:18:29.460535 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:18:29.460497 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:18:32.131982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:32.129432 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:32.350536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:32.350476 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:32.441102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:32.440963 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:32.609361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:32.608454 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:18:32.968589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:32.967723 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:33.059492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.059447 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:33.060153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.060116 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:33.093553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.093510 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:33.132266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.132235 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:33.133481 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.133459 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:18:33.136935 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.132174 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:33.177483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.177431 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:33.369474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.361732 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:33.461362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.444496 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:18:33.480495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.462988 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:33.481374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.481325 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:18:33.558361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.534028 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 04 01:18:33.752963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.752821 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:18:33.781778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:33.781745 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:18:34.063357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.054623 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:34.116606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.099709 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:34.116606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.100838 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:18:34.116606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.104630 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 04 01:18:34.160357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.118792 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:34.372412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.370242 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:34.402363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.401496 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:34.405893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.405846 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:34.406379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.406325 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:18:34.499360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.488978 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:34.511368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.510297 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:18:34.551400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.523922 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:18:34.712516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.709599 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:18:34.733752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.733721 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:18:34.735090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.735071 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:18:34.736114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.736097 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:18:34.754525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:34.754427 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:35.069360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.034473 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:35.160131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.088414 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:35.174417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.105105 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:35.174417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.134681 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:18:35.574360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.544117 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:35.574360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.545593 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:35.574360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.545904 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:35.574360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.546177 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:35.574360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.558538 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:18:35.824124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:35.824068 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:36.596664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.596612 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:36.597606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.597563 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:36.598232 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.598199 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:36.602356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.599032 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:36.639292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.637399 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:18:36.780367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.661144 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:36.788278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.665827 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:18:36.788977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.665856 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:18:36.814244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.768447 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:18:36.815646 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.814521 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:36.937538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:36.937421 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:37.629360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.620282 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:37.629360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.620616 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:37.629360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.624739 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:37.641454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.641406 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:37.741887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.739446 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:37.768285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.764489 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:37.794411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.788402 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:37.794411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.789378 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:37.817693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.815443 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:37.817693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.816517 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:37.941547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:37.941442 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.627357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.626497 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.636359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.629444 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.665391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.630723 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:38.666008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.647626 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.715183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.715136 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.803153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.803106 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.804654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.804366 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.805614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.805586 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:18:38.820374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.820290 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.821977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.821943 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:18:38.855628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.855584 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:38.855929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.855896 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.857389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.857362 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:38.857568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.857542 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.904451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.893379 4312 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:38.904451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.903552 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:38.959415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:38.943591 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:39.315379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.310423 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:39.687448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.683427 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:39.716361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.709240 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:39.716361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.712641 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:39.741403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.721878 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:39.804670 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.804614 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:39.807243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.807209 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:39.807834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.807791 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:18:39.830852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.830803 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.175212 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.831163 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.176320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.932560 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Nov 04 01:18:40.176757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:39.962834 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:40.448421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.438805 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 04 01:18:40.502360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.500273 4312 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-hkx52 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.6:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:40.525610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.500486 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hkx52" podUID="c2c762ba-d7cd-42e2-8410-1bf6fe2408a3" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.133.0.6:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.691374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.688035 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.817361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.806457 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.817361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.806850 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:40.817361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.807574 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.817361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.808731 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:40.817361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.808776 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.835359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.818931 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:40.938998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.930477 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:40.938998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:40.930558 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.159588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.159492 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.184943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.183218 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.218558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.217507 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:41.609358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.595811 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:41.609358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.595884 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-4l6sk" podUID="33e3cbdc-f101-49ff-953e-c46a13d2658e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.644373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.639806 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.814360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.811385 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:41.838059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.838009 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.843740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.843696 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:41.856677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.856631 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.860485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.860398 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:41.863804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.860734 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:18:41.864072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:41.860766 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:42.228543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.183760 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:42.410311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.184016 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:42.410311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.242016 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:18:42.499723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.499627 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8"] Nov 04 01:18:42.523698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.523661 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2"] Nov 04 01:18:42.816505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.816470 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:18:42.870356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.849646 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:42.870356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.850023 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:18:42.875405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.875161 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:42.876172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.876005 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:42.877578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.876552 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:42.877578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:42.876802 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.532898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.532729 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:18:43.539357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.533240 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.539357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.537618 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:18:43.574580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.574533 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.845466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.845415 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.883587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.855445 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:18:43.883984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.883965 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:18:43.884891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.855529 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.885893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.855744 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:18:43.889356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.883824 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.891673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.883885 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.892170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.883861 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:43.939934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:43.938595 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:44.388137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.376872 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:18:44.630358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.612834 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:44.659417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.645842 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:18:44.789359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.752885 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:18:44.789359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.768734 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:44.858464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.858416 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:44.949526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.949431 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:44.950117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:44.950085 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:45.008406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:45.007420 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:18:45.008406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:45.007711 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:18:45.088988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:45.088956 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:18:45.143914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:45.143885 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:18:45.244220 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:45.236748 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:18:45.322676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:45.322651 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:18:47.501076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:47.501025 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" podUID="204d1390-12f5-45e8-a024-c55df88b9605" containerName="payload-processing" containerID="cri-o://b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf" gracePeriod=25 Nov 04 01:18:47.524681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:47.524637 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" podUID="617f8d0e-7252-4d6c-859b-74168c0ca47f" containerName="payload-pre-processing" containerID="cri-o://0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143" gracePeriod=25 Nov 04 01:18:47.990271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:47.990246 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:47.993962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:47.993940 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:48.032888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.032854 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204d1390-12f5-45e8-a024-c55df88b9605-kube-api-access-k9x85\" (UniqueName: \"kubernetes.io/projected/204d1390-12f5-45e8-a024-c55df88b9605-kube-api-access-k9x85\") pod \"204d1390-12f5-45e8-a024-c55df88b9605\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " Nov 04 01:18:48.033134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.033114 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617f8d0e-7252-4d6c-859b-74168c0ca47f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617f8d0e-7252-4d6c-859b-74168c0ca47f-plugins-config-volume\") pod \"617f8d0e-7252-4d6c-859b-74168c0ca47f\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " Nov 04 01:18:48.033246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.033197 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204d1390-12f5-45e8-a024-c55df88b9605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204d1390-12f5-45e8-a024-c55df88b9605-plugins-config-volume\") pod \"204d1390-12f5-45e8-a024-c55df88b9605\" (UID: \"204d1390-12f5-45e8-a024-c55df88b9605\") " Nov 04 01:18:48.033246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.033233 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617f8d0e-7252-4d6c-859b-74168c0ca47f-kube-api-access-jnjw7\" (UniqueName: \"kubernetes.io/projected/617f8d0e-7252-4d6c-859b-74168c0ca47f-kube-api-access-jnjw7\") pod \"617f8d0e-7252-4d6c-859b-74168c0ca47f\" (UID: \"617f8d0e-7252-4d6c-859b-74168c0ca47f\") " Nov 04 01:18:48.034886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.034858 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617f8d0e-7252-4d6c-859b-74168c0ca47f-plugins-config-volume" pod "617f8d0e-7252-4d6c-859b-74168c0ca47f" (UID: "617f8d0e-7252-4d6c-859b-74168c0ca47f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:18:48.035242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.034882 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204d1390-12f5-45e8-a024-c55df88b9605-plugins-config-volume" pod "204d1390-12f5-45e8-a024-c55df88b9605" (UID: "204d1390-12f5-45e8-a024-c55df88b9605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:18:48.036839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.036806 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617f8d0e-7252-4d6c-859b-74168c0ca47f-kube-api-access-jnjw7" pod "617f8d0e-7252-4d6c-859b-74168c0ca47f" (UID: "617f8d0e-7252-4d6c-859b-74168c0ca47f"). InnerVolumeSpecName "kube-api-access-jnjw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:48.044909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.044873 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204d1390-12f5-45e8-a024-c55df88b9605-kube-api-access-k9x85" pod "204d1390-12f5-45e8-a024-c55df88b9605" (UID: "204d1390-12f5-45e8-a024-c55df88b9605"). InnerVolumeSpecName "kube-api-access-k9x85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:48.052598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.052568 4312 generic.go:356] "Generic (PLEG): container finished" podID="204d1390-12f5-45e8-a024-c55df88b9605" containerID="b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf" exitCode=0 Nov 04 01:18:48.052809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.052770 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" event={"ID":"204d1390-12f5-45e8-a024-c55df88b9605","Type":"ContainerDied","Data":"b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf"} Nov 04 01:18:48.052919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.052902 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" event={"ID":"204d1390-12f5-45e8-a024-c55df88b9605","Type":"ContainerDied","Data":"3da9a4f544607d0e8be34163e341dac94a755b1c93fb9191b4b73800643a2b74"} Nov 04 01:18:48.052984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.052954 4312 scope.go:122] "RemoveContainer" containerID="b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf" Nov 04 01:18:48.053153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.053137 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8" Nov 04 01:18:48.055714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.055687 4312 generic.go:356] "Generic (PLEG): container finished" podID="617f8d0e-7252-4d6c-859b-74168c0ca47f" containerID="0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143" exitCode=0 Nov 04 01:18:48.055817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.055765 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" event={"ID":"617f8d0e-7252-4d6c-859b-74168c0ca47f","Type":"ContainerDied","Data":"0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143"} Nov 04 01:18:48.055817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.055809 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" event={"ID":"617f8d0e-7252-4d6c-859b-74168c0ca47f","Type":"ContainerDied","Data":"a0ffb502c5ad512e8e768764e4206ad5cb0c6b0f37b57026ca74b910b2d4578a"} Nov 04 01:18:48.055933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.055880 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2" Nov 04 01:18:48.067524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.067499 4312 scope.go:122] "RemoveContainer" containerID="b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf" Nov 04 01:18:48.067862 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:18:48.067840 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf\": container with ID starting with b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf not found: ID does not exist" containerID="b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf" Nov 04 01:18:48.067968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.067868 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf"} err="failed to get container status \"b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf\": rpc error: code = NotFound desc = could not find container \"b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf\": container with ID starting with b82969f6a46d3bfbbc3d62df471a34ae605bf01aa1129b0ced9df27205031daf not found: ID does not exist" Nov 04 01:18:48.067968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.067887 4312 scope.go:122] "RemoveContainer" containerID="0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143" Nov 04 01:18:48.081810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.081745 4312 scope.go:122] "RemoveContainer" containerID="0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143" Nov 04 01:18:48.082083 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:18:48.082059 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143\": container with ID starting with 0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143 not found: ID does not exist" containerID="0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143" Nov 04 01:18:48.082145 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.082094 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143"} err="failed to get container status \"0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143\": rpc error: code = NotFound desc = could not find container \"0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143\": container with ID starting with 0af2f825805d69b452d5cf6d896276b93a3ac03632fef262230b5fa2ce681143 not found: ID does not exist" Nov 04 01:18:48.091924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.091904 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8"] Nov 04 01:18:48.097325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.097302 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cc22d46-786f8f8fcb-7x7b8"] Nov 04 01:18:48.111562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.111538 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204d1390-12f5-45e8-a024-c55df88b9605" path="/var/lib/kubelet/pods/204d1390-12f5-45e8-a024-c55df88b9605/volumes" Nov 04 01:18:48.113962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.113940 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2"] Nov 04 01:18:48.120447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.120415 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cc22d46-76c7f5d479-j7zn2"] Nov 04 01:18:48.136384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.136365 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9x85\" (UniqueName: \"kubernetes.io/projected/204d1390-12f5-45e8-a024-c55df88b9605-kube-api-access-k9x85\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:48.136384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.136385 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617f8d0e-7252-4d6c-859b-74168c0ca47f-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:48.136505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.136395 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204d1390-12f5-45e8-a024-c55df88b9605-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:48.136505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:48.136409 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnjw7\" (UniqueName: \"kubernetes.io/projected/617f8d0e-7252-4d6c-859b-74168c0ca47f-kube-api-access-jnjw7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:50.182354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:50.181737 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617f8d0e-7252-4d6c-859b-74168c0ca47f" path="/var/lib/kubelet/pods/617f8d0e-7252-4d6c-859b-74168c0ca47f/volumes" Nov 04 01:18:52.238354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.237770 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n"] Nov 04 01:18:52.256420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.238971 4312 cpu_manager.go:414] "containerMap: removing container" podUID="204d1390-12f5-45e8-a024-c55df88b9605" containerName="payload-processing" Nov 04 01:18:52.256420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.238998 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204d1390-12f5-45e8-a024-c55df88b9605" containerName="payload-processing" Nov 04 01:18:52.256420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.239038 4312 cpu_manager.go:414] "containerMap: removing container" podUID="617f8d0e-7252-4d6c-859b-74168c0ca47f" containerName="payload-pre-processing" Nov 04 01:18:52.256420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.239046 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617f8d0e-7252-4d6c-859b-74168c0ca47f" containerName="payload-pre-processing" Nov 04 01:18:52.256420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.239232 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="204d1390-12f5-45e8-a024-c55df88b9605" containerName="payload-processing" Nov 04 01:18:52.256420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.239247 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="617f8d0e-7252-4d6c-859b-74168c0ca47f" containerName="payload-pre-processing" Nov 04 01:18:52.256843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.256614 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n"] Nov 04 01:18:52.259562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.257765 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.263368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.263287 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cca527b7-gw-openshift-default-dockercfg-ht6gn\"" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385014 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcmk\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-kube-api-access-zmcmk\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385182 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-socket\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385269 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166b36-1e15-448c-a77e-fb490258f463-istiod-ca-cert\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385323 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-envoy\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385435 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166b36-1e15-448c-a77e-fb490258f463-istio-podinfo\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385542 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-credential-socket\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385678 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-certs\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385746 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-data\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.400368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.385904 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-istio-token\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.504680 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166b36-1e15-448c-a77e-fb490258f463-istiod-ca-cert\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.504762 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-envoy\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.504851 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166b36-1e15-448c-a77e-fb490258f463-istio-podinfo\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.504930 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-credential-socket\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.505034 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-certs\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.505095 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-data\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.505198 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-istio-token\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.512484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.505262 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcmk\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-kube-api-access-zmcmk\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.550364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.549535 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166b36-1e15-448c-a77e-fb490258f463-istio-podinfo\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.558364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.558051 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166b36-1e15-448c-a77e-fb490258f463-istiod-ca-cert\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.565503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.559956 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-credential-socket\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.565503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.560224 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-certs\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.583361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.582394 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-socket\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.630274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.620366 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-data\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.660791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.660761 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-socket\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.663889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.663853 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-envoy\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.666897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.666766 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-istio-token\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.669815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.669787 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcmk\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-kube-api-access-zmcmk\") pod \"e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.696663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.696629 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:52.993633 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:18:52.993602 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe166b36_1e15_448c_a77e_fb490258f463.slice/crio-2c6a68f1a45059a9d7f1d4685032a7cdaf47ad9cf1706cb0e4922585c22e1b11 WatchSource:0}: Error finding container 2c6a68f1a45059a9d7f1d4685032a7cdaf47ad9cf1706cb0e4922585c22e1b11: Status 404 returned error can't find the container with id 2c6a68f1a45059a9d7f1d4685032a7cdaf47ad9cf1706cb0e4922585c22e1b11 Nov 04 01:18:52.993905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.993883 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n"] Nov 04 01:18:52.996640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.996606 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:18:52.996724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.996673 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:18:52.996724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:52.996702 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:18:53.264357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:53.264310 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" event={"ID":"fe166b36-1e15-448c-a77e-fb490258f463","Type":"ContainerStarted","Data":"738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35"} Nov 04 01:18:53.264876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:53.264852 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" event={"ID":"fe166b36-1e15-448c-a77e-fb490258f463","Type":"ContainerStarted","Data":"2c6a68f1a45059a9d7f1d4685032a7cdaf47ad9cf1706cb0e4922585c22e1b11"} Nov 04 01:18:53.297359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:53.297290 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" podStartSLOduration=1.297275939 podStartE2EDuration="1.297275939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:53.293036311 +0000 UTC m=+965.315968329" watchObservedRunningTime="2026-11-04 01:18:53.297275939 +0000 UTC m=+965.320207946" Nov 04 01:18:53.698671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:53.698583 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:54.702990 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:54.702956 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:55.281364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.281321 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:55.282351 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.282313 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:18:55.348422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.348349 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd"] Nov 04 01:18:55.355005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.354978 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:55.363385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.357406 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j"] Nov 04 01:18:55.363385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.362070 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cca527b7\"" Nov 04 01:18:55.364273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.364248 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:55.369101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.369078 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cca527b7-dockercfg-9h7zn\"" Nov 04 01:18:55.371424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.371382 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd"] Nov 04 01:18:55.381433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.381408 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j"] Nov 04 01:18:55.532054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.531951 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b898d\" (UniqueName: \"kubernetes.io/projected/47a0525c-e5c1-4562-aba3-644085829b40-kube-api-access-b898d\") pod \"payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:55.532500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.532453 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqgx\" (UniqueName: \"kubernetes.io/projected/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-kube-api-access-fkqgx\") pod \"payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:55.545398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.532711 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:55.545788 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.545749 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a0525c-e5c1-4562-aba3-644085829b40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:55.650356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.649646 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b898d\" (UniqueName: \"kubernetes.io/projected/47a0525c-e5c1-4562-aba3-644085829b40-kube-api-access-b898d\") pod \"payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:55.650356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.649882 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqgx\" (UniqueName: \"kubernetes.io/projected/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-kube-api-access-fkqgx\") pod \"payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:55.650356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.649963 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:55.650356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.650078 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a0525c-e5c1-4562-aba3-644085829b40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:55.656353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.651141 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a0525c-e5c1-4562-aba3-644085829b40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:55.656496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.656390 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:55.720228 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.720189 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqgx\" (UniqueName: \"kubernetes.io/projected/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-kube-api-access-fkqgx\") pod \"payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:55.721421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:55.721394 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b898d\" (UniqueName: \"kubernetes.io/projected/47a0525c-e5c1-4562-aba3-644085829b40-kube-api-access-b898d\") pod \"payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:56.005900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:56.005711 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:56.028789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:56.017051 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:56.381621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:56.381587 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j"] Nov 04 01:18:56.395915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:56.395887 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd"] Nov 04 01:18:57.300611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.300572 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" event={"ID":"47a0525c-e5c1-4562-aba3-644085829b40","Type":"ContainerStarted","Data":"17a9f96a739f19c52df086a3e74ab17090aa5ee3ddd70bbbb1968aeeec83e3dc"} Nov 04 01:18:57.301064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.300638 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" event={"ID":"47a0525c-e5c1-4562-aba3-644085829b40","Type":"ContainerStarted","Data":"2ff1043a4a7249ce615d2410e4829359bc52c4de3d9be1947aeee1df258b2738"} Nov 04 01:18:57.301064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.300665 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:18:57.302369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.302325 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" event={"ID":"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03","Type":"ContainerStarted","Data":"1478d82fba7157d68b5082af28fc58c3ccf7e97442369470c4150af358d888a7"} Nov 04 01:18:57.302491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.302379 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" event={"ID":"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03","Type":"ContainerStarted","Data":"7387d17602c765132006f7bcf9be7ca8b96580658ac71ec0507c5c9ec8d6f77e"} Nov 04 01:18:57.302491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.302426 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:18:57.322789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.322745 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" podStartSLOduration=2.3227323269999998 podStartE2EDuration="2.322732327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:57.318181832 +0000 UTC m=+969.341113857" watchObservedRunningTime="2026-11-04 01:18:57.322732327 +0000 UTC m=+969.345664333" Nov 04 01:18:57.339672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:57.339621 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" podStartSLOduration=2.339607998 podStartE2EDuration="2.339607998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:57.33570875 +0000 UTC m=+969.358640799" watchObservedRunningTime="2026-11-04 01:18:57.339607998 +0000 UTC m=+969.362540005" Nov 04 01:18:58.826185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:18:58.826146 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s"] Nov 04 01:18:59.474463 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:18:59.474417 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:19:01.678407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.672900 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6"] Nov 04 01:19:01.704353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.701976 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.758362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.751268 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-029e8c-openshift-default-dockercfg-g5zhv\"" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.863783 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-token\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.863893 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-certs\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.863999 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-data\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.864129 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-credential-socket\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.864176 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-podinfo\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.864200 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6th9n\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-kube-api-access-6th9n\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.864274 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f775a2-6d6c-422c-be49-5eca668d52c8-istiod-ca-cert\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.864319 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-envoy\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.878687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.864479 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-socket\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967358 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-envoy\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967466 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-socket\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967671 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-token\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967718 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-certs\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967799 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-data\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967892 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-credential-socket\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967936 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-podinfo\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.967963 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6th9n\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-kube-api-access-6th9n\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.968019 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f775a2-6d6c-422c-be49-5eca668d52c8-istiod-ca-cert\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.973400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.968965 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f775a2-6d6c-422c-be49-5eca668d52c8-istiod-ca-cert\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.995363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.988726 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-socket\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.995363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.989220 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-certs\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.995363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.989413 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-data\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:01.995363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:01.989567 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-credential-socket\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:02.036285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.036233 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6"] Nov 04 01:19:02.196570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.196401 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-podinfo\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:02.242267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.242179 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-token\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:02.248600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.248567 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-envoy\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:02.284651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.277961 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6th9n\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-kube-api-access-6th9n\") pod \"e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:02.288408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.288328 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:02.632516 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:02.632480 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f775a2_6d6c_422c_be49_5eca668d52c8.slice/crio-0eea3e187618b74df9577ea789c3bb6526cf3c0b299ccb5ef2b47381ab1ebe0f WatchSource:0}: Error finding container 0eea3e187618b74df9577ea789c3bb6526cf3c0b299ccb5ef2b47381ab1ebe0f: Status 404 returned error can't find the container with id 0eea3e187618b74df9577ea789c3bb6526cf3c0b299ccb5ef2b47381ab1ebe0f Nov 04 01:19:02.632958 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.632933 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6"] Nov 04 01:19:02.635232 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.635205 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:02.635295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.635266 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:02.635346 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:02.635294 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:03.136129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.136087 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj"] Nov 04 01:19:03.142610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.142580 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.146317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.146291 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cca527b7-serving-cert\"" Nov 04 01:19:03.149454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.149430 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj"] Nov 04 01:19:03.296367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.294387 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-maas-api-tls\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.296367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.294516 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e2e578-6bf9-4bc9-91ff-b93b154ee100-openshift-service-ca\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.296367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.294623 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-metrics-certs\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.296367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.294691 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ltm\" (UniqueName: \"kubernetes.io/projected/64e2e578-6bf9-4bc9-91ff-b93b154ee100-kube-api-access-v4ltm\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.403028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.398720 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-maas-api-tls\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.403028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.398827 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e2e578-6bf9-4bc9-91ff-b93b154ee100-openshift-service-ca\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.403028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.398935 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-metrics-certs\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.403028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.399009 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ltm\" (UniqueName: \"kubernetes.io/projected/64e2e578-6bf9-4bc9-91ff-b93b154ee100-kube-api-access-v4ltm\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.403028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.402278 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e2e578-6bf9-4bc9-91ff-b93b154ee100-openshift-service-ca\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.406509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.406418 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-metrics-certs\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.409889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.409863 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-maas-api-tls\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.414080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.414059 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ltm\" (UniqueName: \"kubernetes.io/projected/64e2e578-6bf9-4bc9-91ff-b93b154ee100-kube-api-access-v4ltm\") pod \"maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.461886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.461855 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:03.578176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.578142 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" event={"ID":"67f775a2-6d6c-422c-be49-5eca668d52c8","Type":"ContainerStarted","Data":"f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc"} Nov 04 01:19:03.578378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.578194 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" event={"ID":"67f775a2-6d6c-422c-be49-5eca668d52c8","Type":"ContainerStarted","Data":"0eea3e187618b74df9577ea789c3bb6526cf3c0b299ccb5ef2b47381ab1ebe0f"} Nov 04 01:19:03.606731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.601136 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" podStartSLOduration=2.60111857 podStartE2EDuration="2.60111857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:03.600737469 +0000 UTC m=+975.623669505" watchObservedRunningTime="2026-11-04 01:19:03.60111857 +0000 UTC m=+975.624050578" Nov 04 01:19:03.845758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.845630 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" podUID="6db301d6-995a-45a2-86ff-1fb9de40e228" containerName="payload-processing" containerID="cri-o://9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758" gracePeriod=25 Nov 04 01:19:03.862709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:03.861723 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj"] Nov 04 01:19:04.295172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.295125 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:04.301157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.301136 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:04.508403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.507812 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:19:04.693139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.592766 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db301d6-995a-45a2-86ff-1fb9de40e228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db301d6-995a-45a2-86ff-1fb9de40e228-plugins-config-volume\") pod \"6db301d6-995a-45a2-86ff-1fb9de40e228\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " Nov 04 01:19:04.693139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.599834 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db301d6-995a-45a2-86ff-1fb9de40e228-plugins-config-volume" pod "6db301d6-995a-45a2-86ff-1fb9de40e228" (UID: "6db301d6-995a-45a2-86ff-1fb9de40e228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:04.693139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.652683 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db301d6-995a-45a2-86ff-1fb9de40e228-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:04.697498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.697143 4312 generic.go:356] "Generic (PLEG): container finished" podID="6db301d6-995a-45a2-86ff-1fb9de40e228" containerID="9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758" exitCode=0 Nov 04 01:19:04.697498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.697269 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" event={"ID":"6db301d6-995a-45a2-86ff-1fb9de40e228","Type":"ContainerDied","Data":"9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758"} Nov 04 01:19:04.697498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.697321 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" event={"ID":"6db301d6-995a-45a2-86ff-1fb9de40e228","Type":"ContainerDied","Data":"ef99c16ca54ed6d1d3f8d9b9497573b3e1568c05fc82f20f5b0620dad38aaefe"} Nov 04 01:19:04.697498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.697368 4312 scope.go:122] "RemoveContainer" containerID="9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758" Nov 04 01:19:04.697498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.697473 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s" Nov 04 01:19:04.701022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.700955 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" event={"ID":"64e2e578-6bf9-4bc9-91ff-b93b154ee100","Type":"ContainerStarted","Data":"93ccd653b0b008266aa2514d571d47177f2bd94b80cd660de9ecd96557b8ce29"} Nov 04 01:19:04.702164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.702140 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:04.703303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.703282 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:19:04.715507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.715489 4312 scope.go:122] "RemoveContainer" containerID="9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758" Nov 04 01:19:04.715827 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:04.715809 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758\": container with ID starting with 9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758 not found: ID does not exist" containerID="9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758" Nov 04 01:19:04.715925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.715836 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758"} err="failed to get container status \"9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758\": rpc error: code = NotFound desc = could not find container \"9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758\": container with ID starting with 9b442af40d8443a3ffbdfa5849df453f1599621720f645d750a7197e5a688758 not found: ID does not exist" Nov 04 01:19:04.753852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.753820 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db301d6-995a-45a2-86ff-1fb9de40e228-kube-api-access-2jw2s\" (UniqueName: \"kubernetes.io/projected/6db301d6-995a-45a2-86ff-1fb9de40e228-kube-api-access-2jw2s\") pod \"6db301d6-995a-45a2-86ff-1fb9de40e228\" (UID: \"6db301d6-995a-45a2-86ff-1fb9de40e228\") " Nov 04 01:19:04.756850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.756804 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db301d6-995a-45a2-86ff-1fb9de40e228-kube-api-access-2jw2s" pod "6db301d6-995a-45a2-86ff-1fb9de40e228" (UID: "6db301d6-995a-45a2-86ff-1fb9de40e228"). InnerVolumeSpecName "kube-api-access-2jw2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:04.857039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.857005 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jw2s\" (UniqueName: \"kubernetes.io/projected/6db301d6-995a-45a2-86ff-1fb9de40e228-kube-api-access-2jw2s\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:04.862652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.861919 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7"] Nov 04 01:19:04.862652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:04.862197 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" podUID="84ce0cb4-d037-465d-b09a-1e6b32d198e1" containerName="maas-api" containerID="cri-o://4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a" gracePeriod=30 Nov 04 01:19:05.030103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.030053 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s"] Nov 04 01:19:05.037783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.037750 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383a864b-745f85645b-rxp4s"] Nov 04 01:19:05.196867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.196837 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:19:05.389488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.388034 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ce0cb4-d037-465d-b09a-1e6b32d198e1-kube-api-access-w5z27\" (UniqueName: \"kubernetes.io/projected/84ce0cb4-d037-465d-b09a-1e6b32d198e1-kube-api-access-w5z27\") pod \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " Nov 04 01:19:05.389488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.388205 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-metrics-certs\") pod \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " Nov 04 01:19:05.389488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.388268 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ce0cb4-d037-465d-b09a-1e6b32d198e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ce0cb4-d037-465d-b09a-1e6b32d198e1-openshift-service-ca\") pod \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " Nov 04 01:19:05.389488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.388557 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-maas-api-tls\") pod \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\" (UID: \"84ce0cb4-d037-465d-b09a-1e6b32d198e1\") " Nov 04 01:19:05.407358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.391742 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ce0cb4-d037-465d-b09a-1e6b32d198e1-openshift-service-ca" pod "84ce0cb4-d037-465d-b09a-1e6b32d198e1" (UID: "84ce0cb4-d037-465d-b09a-1e6b32d198e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:05.434707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.415778 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ce0cb4-d037-465d-b09a-1e6b32d198e1-kube-api-access-w5z27" pod "84ce0cb4-d037-465d-b09a-1e6b32d198e1" (UID: "84ce0cb4-d037-465d-b09a-1e6b32d198e1"). InnerVolumeSpecName "kube-api-access-w5z27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:05.453713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.441458 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-maas-api-tls" pod "84ce0cb4-d037-465d-b09a-1e6b32d198e1" (UID: "84ce0cb4-d037-465d-b09a-1e6b32d198e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:05.453713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.441540 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-metrics-certs" pod "84ce0cb4-d037-465d-b09a-1e6b32d198e1" (UID: "84ce0cb4-d037-465d-b09a-1e6b32d198e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:05.496402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.491799 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5z27\" (UniqueName: \"kubernetes.io/projected/84ce0cb4-d037-465d-b09a-1e6b32d198e1-kube-api-access-w5z27\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:05.496402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.491838 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:05.496402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.491858 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ce0cb4-d037-465d-b09a-1e6b32d198e1-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:05.496402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.491876 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ce0cb4-d037-465d-b09a-1e6b32d198e1-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:05.757042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.756956 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" event={"ID":"64e2e578-6bf9-4bc9-91ff-b93b154ee100","Type":"ContainerStarted","Data":"14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f"} Nov 04 01:19:05.757467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.757443 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:05.758885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.758851 4312 generic.go:356] "Generic (PLEG): container finished" podID="84ce0cb4-d037-465d-b09a-1e6b32d198e1" containerID="4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a" exitCode=0 Nov 04 01:19:05.758979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.758891 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" Nov 04 01:19:05.759274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.759249 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" event={"ID":"84ce0cb4-d037-465d-b09a-1e6b32d198e1","Type":"ContainerDied","Data":"4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a"} Nov 04 01:19:05.759377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.759293 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7" event={"ID":"84ce0cb4-d037-465d-b09a-1e6b32d198e1","Type":"ContainerDied","Data":"95a13b7d96390ca44ba964cb774011d4f044a6dcb992d51744d36e88d448342d"} Nov 04 01:19:05.759377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.759317 4312 scope.go:122] "RemoveContainer" containerID="4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a" Nov 04 01:19:05.776810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.776787 4312 scope.go:122] "RemoveContainer" containerID="4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a" Nov 04 01:19:05.777346 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:05.777303 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a\": container with ID starting with 4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a not found: ID does not exist" containerID="4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a" Nov 04 01:19:05.777446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.777379 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a"} err="failed to get container status \"4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a\": rpc error: code = NotFound desc = could not find container \"4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a\": container with ID starting with 4eed6afb5ddd29299405200735b24a313dddfb12e9fb6d55c2e8edeea8f5339a not found: ID does not exist" Nov 04 01:19:05.780647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.780607 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" podStartSLOduration=1.971600183 podStartE2EDuration="2.780593475s" totalImagesPullingTime="808.993292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:05.777051668 +0000 UTC m=+977.799983713" watchObservedRunningTime="2026-11-04 01:19:05.780593475 +0000 UTC m=+977.803525525" Nov 04 01:19:05.802849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.802812 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7"] Nov 04 01:19:05.809101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:05.809069 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-383a864b-fb47bb76-hjcl7"] Nov 04 01:19:06.110734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:06.110688 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db301d6-995a-45a2-86ff-1fb9de40e228" path="/var/lib/kubelet/pods/6db301d6-995a-45a2-86ff-1fb9de40e228/volumes" Nov 04 01:19:06.111261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:06.111245 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ce0cb4-d037-465d-b09a-1e6b32d198e1" path="/var/lib/kubelet/pods/84ce0cb4-d037-465d-b09a-1e6b32d198e1/volumes" Nov 04 01:19:06.405914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:06.405827 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f"] Nov 04 01:19:06.406323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:06.406075 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" podUID="0139988f-7aa7-4de8-8209-a0a5cda050e2" containerName="istio-proxy" containerID="cri-o://9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891" gracePeriod=30 Nov 04 01:19:07.616991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.616959 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r"] Nov 04 01:19:07.618150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.618121 4312 cpu_manager.go:414] "containerMap: removing container" podUID="84ce0cb4-d037-465d-b09a-1e6b32d198e1" containerName="maas-api" Nov 04 01:19:07.618150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.618147 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ce0cb4-d037-465d-b09a-1e6b32d198e1" containerName="maas-api" Nov 04 01:19:07.618418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.618182 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6db301d6-995a-45a2-86ff-1fb9de40e228" containerName="payload-processing" Nov 04 01:19:07.618418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.618190 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db301d6-995a-45a2-86ff-1fb9de40e228" containerName="payload-processing" Nov 04 01:19:07.618529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.618423 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ce0cb4-d037-465d-b09a-1e6b32d198e1" containerName="maas-api" Nov 04 01:19:07.618529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.618444 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db301d6-995a-45a2-86ff-1fb9de40e228" containerName="payload-processing" Nov 04 01:19:07.624786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.624761 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.643254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.642992 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r"] Nov 04 01:19:07.643254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.643199 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8ab7c587-openshift-default-dockercfg-qmnt9\"" Nov 04 01:19:07.707523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.707495 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z579r\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-kube-api-access-z579r\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.707665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.707617 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-podinfo\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.707724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.707685 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-data\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.707784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.707767 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istiod-ca-cert\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.707858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.707836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-credential-socket\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.707929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.707907 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-envoy\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.708031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.708017 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-socket\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.708087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.708076 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-certs\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.708163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.708145 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-token\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.785809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.785773 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w"] Nov 04 01:19:07.790639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.790615 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:07.809378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809344 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-podinfo\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809411 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-data\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809503 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istiod-ca-cert\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809543 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-credential-socket\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809595 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-envoy\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809729 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-socket\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809802 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-data\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809809 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-certs\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809876 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-token\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.809977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809963 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z579r\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-kube-api-access-z579r\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.810031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.809982 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-credential-socket\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.810084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.810039 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-certs\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.810228 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.810199 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istiod-ca-cert\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.810531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.810509 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-socket\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.812448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.812415 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-envoy\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.813966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.813325 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-podinfo\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.825002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.824976 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w"] Nov 04 01:19:07.825699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.825673 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z579r\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-kube-api-access-z579r\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.829448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.827125 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-token\") pod \"e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:07.880379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.880295 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-029e8c-dockercfg-ss5sw\"" Nov 04 01:19:07.880379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.880295 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-029e8c\"" Nov 04 01:19:07.913181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.913144 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948j7\" (UniqueName: \"kubernetes.io/projected/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-kube-api-access-948j7\") pod \"payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:07.913376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.913233 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:07.950093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:07.950061 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:08.039357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.030864 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948j7\" (UniqueName: \"kubernetes.io/projected/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-kube-api-access-948j7\") pod \"payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:08.039357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.030975 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:08.039357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.032423 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:08.188448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.185037 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-948j7\" (UniqueName: \"kubernetes.io/projected/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-kube-api-access-948j7\") pod \"payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:08.409131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.408833 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:08.434663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.433596 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:19:08.434663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.433683 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:19:08.681613 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:08.681582 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d0b0f1_63aa_4d98_a86b_e19c310cf369.slice/crio-72d55d434980416846304d3ea1c3f8c3a13ba78093a885ba11c8ac9afa400833 WatchSource:0}: Error finding container 72d55d434980416846304d3ea1c3f8c3a13ba78093a885ba11c8ac9afa400833: Status 404 returned error can't find the container with id 72d55d434980416846304d3ea1c3f8c3a13ba78093a885ba11c8ac9afa400833 Nov 04 01:19:08.683031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.683008 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r"] Nov 04 01:19:08.684531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.684499 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:08.684608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.684567 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:08.684608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.684597 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:08.738214 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:08.738183 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9ed4bb_c6c5_4a50_8283_1cab9b3f1898.slice/crio-460e1cf6dd0d3e9c129b699e550a9b1f2bda99b900005e98891de5ebd53b4bdc WatchSource:0}: Error finding container 460e1cf6dd0d3e9c129b699e550a9b1f2bda99b900005e98891de5ebd53b4bdc: Status 404 returned error can't find the container with id 460e1cf6dd0d3e9c129b699e550a9b1f2bda99b900005e98891de5ebd53b4bdc Nov 04 01:19:08.738907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.738889 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w"] Nov 04 01:19:08.856937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.856906 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" event={"ID":"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898","Type":"ContainerStarted","Data":"2e1dea7aadb697ba14e1cfcce8dcf2b95f45d53e2f9f6855726b1f24b9e3f95a"} Nov 04 01:19:08.857102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.856963 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" event={"ID":"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898","Type":"ContainerStarted","Data":"460e1cf6dd0d3e9c129b699e550a9b1f2bda99b900005e98891de5ebd53b4bdc"} Nov 04 01:19:08.857102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.856993 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:08.859194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.859166 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" event={"ID":"05d0b0f1-63aa-4d98-a86b-e19c310cf369","Type":"ContainerStarted","Data":"cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451"} Nov 04 01:19:08.859320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.859220 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" event={"ID":"05d0b0f1-63aa-4d98-a86b-e19c310cf369","Type":"ContainerStarted","Data":"72d55d434980416846304d3ea1c3f8c3a13ba78093a885ba11c8ac9afa400833"} Nov 04 01:19:08.876045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.875992 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" podStartSLOduration=1.875979563 podStartE2EDuration="1.875979563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:08.87539105 +0000 UTC m=+980.898323077" watchObservedRunningTime="2026-11-04 01:19:08.875979563 +0000 UTC m=+980.898911569" Nov 04 01:19:08.901648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.901602 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" podStartSLOduration=1.901588823 podStartE2EDuration="1.901588823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:08.897716314 +0000 UTC m=+980.920648342" watchObservedRunningTime="2026-11-04 01:19:08.901588823 +0000 UTC m=+980.924520829" Nov 04 01:19:08.950814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:08.950744 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:09.984886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:09.984859 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:10.286647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:10.286618 4312 patch_prober.go:29] interesting pod/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:19:10.286842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:10.286674 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" podUID="0139988f-7aa7-4de8-8209-a0a5cda050e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:19:10.946843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:10.945162 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:10.956907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:10.951560 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:11.722271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.722242 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:19:11.921002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.920708 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-podinfo\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.921002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.920782 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-data\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.921002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.920897 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-kube-api-access-mvmwh\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-kube-api-access-mvmwh\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.921311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.921172 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-credential-socket\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.921311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.921244 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-token\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-token\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.921311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.921302 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-socket\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.955021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.921406 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-certs\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.955021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.921508 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0139988f-7aa7-4de8-8209-a0a5cda050e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0139988f-7aa7-4de8-8209-a0a5cda050e2-istiod-ca-cert\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.955021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.921639 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-envoy\") pod \"0139988f-7aa7-4de8-8209-a0a5cda050e2\" (UID: \"0139988f-7aa7-4de8-8209-a0a5cda050e2\") " Nov 04 01:19:11.955021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.933767 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-credential-socket" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:11.959456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.958835 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-socket" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:11.987467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.987410 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-certs" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:11.989222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.989121 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-data" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:12.184388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:11.997577 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0139988f-7aa7-4de8-8209-a0a5cda050e2-istiod-ca-cert" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:12.328708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.034549 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-kube-api-access-mvmwh" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "kube-api-access-mvmwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:12.340364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.324462 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-podinfo" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:19:12.340364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.326178 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-token" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:12.340364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.320730 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:12.340364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.336620 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:12.340364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.336647 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:12.340364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.336669 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:12.340364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.090534 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:12.898484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.897873 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:19:12.998964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:12.998910 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:13.053520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.053481 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:13.059364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.056180 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-envoy" pod "0139988f-7aa7-4de8-8209-a0a5cda050e2" (UID: "0139988f-7aa7-4de8-8209-a0a5cda050e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:13.092632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.092604 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:13.092871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.092852 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0139988f-7aa7-4de8-8209-a0a5cda050e2-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:13.093097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.093081 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:13.093245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.093229 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0139988f-7aa7-4de8-8209-a0a5cda050e2-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:13.093470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.093451 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvmwh\" (UniqueName: \"kubernetes.io/projected/0139988f-7aa7-4de8-8209-a0a5cda050e2-kube-api-access-mvmwh\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:13.269221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.269129 4312 generic.go:356] "Generic (PLEG): container finished" podID="0139988f-7aa7-4de8-8209-a0a5cda050e2" containerID="9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891" exitCode=0 Nov 04 01:19:13.284387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.275728 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" Nov 04 01:19:13.430971 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:13.430892 4312 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.046s" Nov 04 01:19:13.431136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.430993 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" event={"ID":"0139988f-7aa7-4de8-8209-a0a5cda050e2","Type":"ContainerDied","Data":"9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891"} Nov 04 01:19:13.431136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.431029 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f" event={"ID":"0139988f-7aa7-4de8-8209-a0a5cda050e2","Type":"ContainerDied","Data":"cb1e23fdbe84404f8f177839a668cf695a03ab32f2f9fbc155754d5d74411e5d"} Nov 04 01:19:13.431136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.431045 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4"] Nov 04 01:19:13.437016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.432165 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0139988f-7aa7-4de8-8209-a0a5cda050e2" containerName="istio-proxy" Nov 04 01:19:13.437016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.432189 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0139988f-7aa7-4de8-8209-a0a5cda050e2" containerName="istio-proxy" Nov 04 01:19:13.458186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.446910 4312 scope.go:122] "RemoveContainer" containerID="9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891" Nov 04 01:19:13.466353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.466292 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0139988f-7aa7-4de8-8209-a0a5cda050e2" containerName="istio-proxy" Nov 04 01:19:13.553462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.549082 4312 scope.go:122] "RemoveContainer" containerID="9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891" Nov 04 01:19:13.553912 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:13.553799 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891\": container with ID starting with 9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891 not found: ID does not exist" containerID="9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891" Nov 04 01:19:13.553912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.553839 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891"} err="failed to get container status \"9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891\": rpc error: code = NotFound desc = could not find container \"9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891\": container with ID starting with 9817acd97fa4a7edaddfb045c1e82e5132b9bca606bbbc9fdb8425c61ed96891 not found: ID does not exist" Nov 04 01:19:13.612356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.599673 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:13.613432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.613221 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8ab7c587\"" Nov 04 01:19:13.613432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.613394 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8ab7c587-dockercfg-pzgcd\"" Nov 04 01:19:13.652426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.628310 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4"] Nov 04 01:19:13.652426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.632438 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx"] Nov 04 01:19:13.652426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.638500 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx"] Nov 04 01:19:13.652426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.638545 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f"] Nov 04 01:19:13.652426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.638641 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:13.657662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.657631 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8f95141e-openshift-default-5f698bb979-s2m8f"] Nov 04 01:19:13.780375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.778696 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:13.780375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.778836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9lb\" (UniqueName: \"kubernetes.io/projected/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-kube-api-access-9g9lb\") pod \"payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:13.780375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.779190 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjtt\" (UniqueName: \"kubernetes.io/projected/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-kube-api-access-ttjtt\") pod \"payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:13.820407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.819632 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:13.922020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.921987 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjtt\" (UniqueName: \"kubernetes.io/projected/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-kube-api-access-ttjtt\") pod \"payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:13.932383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.922651 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:13.932778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.932752 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:13.933021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.933004 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9lb\" (UniqueName: \"kubernetes.io/projected/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-kube-api-access-9g9lb\") pod \"payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:13.933434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.923659 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:13.934414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.934393 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:13.969367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.967581 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjtt\" (UniqueName: \"kubernetes.io/projected/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-kube-api-access-ttjtt\") pod \"payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:14.009370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:13.999566 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:19:14.009370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.000162 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:19:14.025982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.024655 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 04 01:19:14.053096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.053055 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9lb\" (UniqueName: \"kubernetes.io/projected/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-kube-api-access-9g9lb\") pod \"payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:14.054525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.054484 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:14.055262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.055246 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:19:14.183412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.160934 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:14.248362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.229627 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:14.328402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.278525 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:14.742984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.742935 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:14.778482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.778438 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:14.804408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.790640 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:19:14.804408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.792226 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:19:14.804800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.792253 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:19:14.805119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.801358 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:14.916885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:14.916804 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0139988f-7aa7-4de8-8209-a0a5cda050e2" path="/var/lib/kubelet/pods/0139988f-7aa7-4de8-8209-a0a5cda050e2/volumes" Nov 04 01:19:15.409372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.405393 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:19:15.426856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.415923 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:15.426856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.416430 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:19:15.426856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.421463 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:19:15.483823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.436413 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:15.809474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.809427 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:19:15.810112 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.810086 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:19:15.834789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.825579 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:19:15.881357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.875030 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:15.902511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.902464 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:15.920143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.920103 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:19:15.920461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.920441 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:19:15.925364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:15.920635 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:19:16.344399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:16.195253 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:19:16.345765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:16.196370 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:19:16.716012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:16.715909 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:19:16.769023 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:16.768978 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 04 01:19:16.919000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:16.918948 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:19:17.000052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:16.999957 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:17.152153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.152107 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:19:17.175138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.175096 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:19:17.670464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.176176 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:19:18.316398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.216217 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:19:18.316398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.216364 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.316398 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:17.584175 4312 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.175s" Nov 04 01:19:18.316398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.293959 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:19:18.316398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.721733 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" podUID="a7f1bc94-14c3-4783-9aad-2efc85a264e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.316398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.877087 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.330296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:17.877179 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:19:18.330296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.179037 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:19:18.402527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.179089 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:19:18.402527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.400481 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 04 01:19:18.431629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.426605 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.431629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.427429 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.488196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.488155 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:19:18.488488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.488456 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.519203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.517866 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" podUID="44760365-91dc-4a74-b176-231e25a8df60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.595468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.583261 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" podUID="44760365-91dc-4a74-b176-231e25a8df60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:18.717117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:18.717030 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.254605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.254542 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k42tl" podUID="870c4be8-527c-43ad-a63f-39dd02718c55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 04 01:19:19.255059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.255010 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk" podUID="ba71124b-dc83-408b-8e5c-1b9abd6fac9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Nov 04 01:19:19.255593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.255571 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ghv26" podUID="61ca9501-204d-4daf-a56b-0794b0f8f6d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.256126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.256095 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.311740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.311702 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.312110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.312083 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.567883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.565071 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 04 01:19:19.594356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.590622 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.594356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.591006 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.606689 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.606652 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:19.607842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.607811 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:19:19.747315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.612410 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 04 01:19:19.907576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.667051 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:19:19.924723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.667111 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:19:19.926366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.667147 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" podUID="a7f1bc94-14c3-4783-9aad-2efc85a264e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Nov 04 01:19:19.927351 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.667221 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" podUID="44760365-91dc-4a74-b176-231e25a8df60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 04 01:19:19.928000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:19.717570 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:20.069265 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:20.067916 4312 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.774s" Nov 04 01:19:20.114422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.094586 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:19:20.133047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.133005 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:19:20.160613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.160573 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.476996 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477091 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477149 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477246 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4"] Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477301 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477362 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477417 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477464 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477520 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:19:20.480368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.477551 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:19:20.554307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.554280 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx"] Nov 04 01:19:20.639589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.639540 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:20.666453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.666408 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:20.667006 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.666981 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:19:20.746163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.746086 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:19:20.781530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.781493 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:20.889241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.889214 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:19:20.892844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.891304 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:20.901623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.891403 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:19:20.901858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.901827 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" event={"ID":"9adbfb87-1c2d-4c81-96bc-d31541bb6beb","Type":"ContainerStarted","Data":"b1732900c96816a35ec5d1af71c979c1cadd4d629f112b80208ef1fb1f067856"} Nov 04 01:19:20.903988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.903968 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:19:20.925177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.923382 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" event={"ID":"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a","Type":"ContainerStarted","Data":"0f6593cd64e7f202c008388b384b1cbcc43eabca33657445d41130c31f968485"} Nov 04 01:19:20.950576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:20.950541 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:21.028148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:21.028118 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:19:21.932779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:21.932736 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" event={"ID":"9adbfb87-1c2d-4c81-96bc-d31541bb6beb","Type":"ContainerStarted","Data":"e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8"} Nov 04 01:19:21.933242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:21.932838 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:21.934654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:21.934621 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" event={"ID":"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a","Type":"ContainerStarted","Data":"81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71"} Nov 04 01:19:21.934766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:21.934687 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:21.955587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:21.955538 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" podStartSLOduration=9.955525872 podStartE2EDuration="9.955525872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:21.951659156 +0000 UTC m=+993.974591178" watchObservedRunningTime="2026-11-04 01:19:21.955525872 +0000 UTC m=+993.978457879" Nov 04 01:19:21.980926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:21.980885 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" podStartSLOduration=9.980870883 podStartE2EDuration="9.980870883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:21.978578952 +0000 UTC m=+994.001510971" watchObservedRunningTime="2026-11-04 01:19:21.980870883 +0000 UTC m=+994.003802924" Nov 04 01:19:24.619408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.619372 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk"] Nov 04 01:19:24.624705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.624674 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.644850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.644804 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8ab7c587-serving-cert\"" Nov 04 01:19:24.662951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.662405 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk"] Nov 04 01:19:24.756360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.748792 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-maas-api-tls\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.756360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.748862 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-openshift-service-ca\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.756360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.749008 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-metrics-certs\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.756360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.749038 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6t5\" (UniqueName: \"kubernetes.io/projected/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-kube-api-access-fg6t5\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.855357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.854770 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-maas-api-tls\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.855357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.854831 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-openshift-service-ca\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.855357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.855057 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-metrics-certs\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.855357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.855096 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6t5\" (UniqueName: \"kubernetes.io/projected/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-kube-api-access-fg6t5\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.864369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.857166 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-openshift-service-ca\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.887033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.886956 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-maas-api-tls\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.891029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.890988 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-metrics-certs\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.895194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.892937 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6t5\" (UniqueName: \"kubernetes.io/projected/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-kube-api-access-fg6t5\") pod \"maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:24.969880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:24.969845 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:25.259541 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:25.259509 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9949f51_5fe1_44d1_b760_0a5a1bb0a18d.slice/crio-18a316b422292d0e884b212473e2908c11bffaa6c7588b5b4aa2551ce9d367ad WatchSource:0}: Error finding container 18a316b422292d0e884b212473e2908c11bffaa6c7588b5b4aa2551ce9d367ad: Status 404 returned error can't find the container with id 18a316b422292d0e884b212473e2908c11bffaa6c7588b5b4aa2551ce9d367ad Nov 04 01:19:25.261197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:25.261167 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk"] Nov 04 01:19:26.134357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:26.119795 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" event={"ID":"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d","Type":"ContainerStarted","Data":"18a316b422292d0e884b212473e2908c11bffaa6c7588b5b4aa2551ce9d367ad"} Nov 04 01:19:27.129621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:27.129585 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" event={"ID":"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d","Type":"ContainerStarted","Data":"56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed"} Nov 04 01:19:27.129852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:27.129821 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:27.151676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:27.151611 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" podStartSLOduration=2.169947129 podStartE2EDuration="3.151596632s" totalImagesPullingTime="981.649503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:27.149425907 +0000 UTC m=+999.172357933" watchObservedRunningTime="2026-11-04 01:19:27.151596632 +0000 UTC m=+999.174528764" Nov 04 01:19:29.459478 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:29.459436 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:19:31.096820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:31.096786 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd"] Nov 04 01:19:31.109274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:31.109239 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j"] Nov 04 01:19:32.966368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:32.963991 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:32.966368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:32.966074 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:33.142774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:33.142742 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:36.098092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.098034 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" podUID="47a0525c-e5c1-4562-aba3-644085829b40" containerName="payload-processing" containerID="cri-o://17a9f96a739f19c52df086a3e74ab17090aa5ee3ddd70bbbb1968aeeec83e3dc" gracePeriod=25 Nov 04 01:19:36.109910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.109846 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" podUID="a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" containerName="payload-pre-processing" containerID="cri-o://1478d82fba7157d68b5082af28fc58c3ccf7e97442369470c4150af358d888a7" gracePeriod=25 Nov 04 01:19:36.301450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.301418 4312 generic.go:356] "Generic (PLEG): container finished" podID="47a0525c-e5c1-4562-aba3-644085829b40" containerID="17a9f96a739f19c52df086a3e74ab17090aa5ee3ddd70bbbb1968aeeec83e3dc" exitCode=0 Nov 04 01:19:36.301608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.301546 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" event={"ID":"47a0525c-e5c1-4562-aba3-644085829b40","Type":"ContainerDied","Data":"17a9f96a739f19c52df086a3e74ab17090aa5ee3ddd70bbbb1968aeeec83e3dc"} Nov 04 01:19:36.304575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.304545 4312 generic.go:356] "Generic (PLEG): container finished" podID="a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" containerID="1478d82fba7157d68b5082af28fc58c3ccf7e97442369470c4150af358d888a7" exitCode=0 Nov 04 01:19:36.304699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.304593 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" event={"ID":"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03","Type":"ContainerDied","Data":"1478d82fba7157d68b5082af28fc58c3ccf7e97442369470c4150af358d888a7"} Nov 04 01:19:36.654511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.654476 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:19:36.691106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.691079 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:19:36.821982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.821945 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a0525c-e5c1-4562-aba3-644085829b40-kube-api-access-b898d\" (UniqueName: \"kubernetes.io/projected/47a0525c-e5c1-4562-aba3-644085829b40-kube-api-access-b898d\") pod \"47a0525c-e5c1-4562-aba3-644085829b40\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " Nov 04 01:19:36.822165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.822081 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a0525c-e5c1-4562-aba3-644085829b40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a0525c-e5c1-4562-aba3-644085829b40-plugins-config-volume\") pod \"47a0525c-e5c1-4562-aba3-644085829b40\" (UID: \"47a0525c-e5c1-4562-aba3-644085829b40\") " Nov 04 01:19:36.822165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.822116 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-plugins-config-volume\") pod \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " Nov 04 01:19:36.822347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.822316 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-kube-api-access-fkqgx\" (UniqueName: \"kubernetes.io/projected/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-kube-api-access-fkqgx\") pod \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\" (UID: \"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03\") " Nov 04 01:19:36.822591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.822565 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a0525c-e5c1-4562-aba3-644085829b40-plugins-config-volume" pod "47a0525c-e5c1-4562-aba3-644085829b40" (UID: "47a0525c-e5c1-4562-aba3-644085829b40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:36.822666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.822565 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-plugins-config-volume" pod "a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" (UID: "a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:36.823064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.823045 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a0525c-e5c1-4562-aba3-644085829b40-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:36.823103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.823067 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:36.824279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.824259 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a0525c-e5c1-4562-aba3-644085829b40-kube-api-access-b898d" pod "47a0525c-e5c1-4562-aba3-644085829b40" (UID: "47a0525c-e5c1-4562-aba3-644085829b40"). InnerVolumeSpecName "kube-api-access-b898d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:36.824378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.824288 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-kube-api-access-fkqgx" pod "a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" (UID: "a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03"). InnerVolumeSpecName "kube-api-access-fkqgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:36.924656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.924623 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqgx\" (UniqueName: \"kubernetes.io/projected/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03-kube-api-access-fkqgx\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:36.924656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:36.924653 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b898d\" (UniqueName: \"kubernetes.io/projected/47a0525c-e5c1-4562-aba3-644085829b40-kube-api-access-b898d\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:37.385804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.385731 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" Nov 04 01:19:37.386160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.386010 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd" event={"ID":"47a0525c-e5c1-4562-aba3-644085829b40","Type":"ContainerDied","Data":"2ff1043a4a7249ce615d2410e4829359bc52c4de3d9be1947aeee1df258b2738"} Nov 04 01:19:37.386160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.386068 4312 scope.go:122] "RemoveContainer" containerID="17a9f96a739f19c52df086a3e74ab17090aa5ee3ddd70bbbb1968aeeec83e3dc" Nov 04 01:19:37.388670 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.388648 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" Nov 04 01:19:37.388867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.388710 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j" event={"ID":"a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03","Type":"ContainerDied","Data":"7387d17602c765132006f7bcf9be7ca8b96580658ac71ec0507c5c9ec8d6f77e"} Nov 04 01:19:37.401416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.401395 4312 scope.go:122] "RemoveContainer" containerID="1478d82fba7157d68b5082af28fc58c3ccf7e97442369470c4150af358d888a7" Nov 04 01:19:37.419994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.419733 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j"] Nov 04 01:19:37.425179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.425153 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cca527b7-75d7bf9679-b9d2j"] Nov 04 01:19:37.435630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.435598 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd"] Nov 04 01:19:37.442635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.442611 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cca527b7-98f87ff68-tk9pd"] Nov 04 01:19:37.699495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.699413 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj"] Nov 04 01:19:37.699664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.699602 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" podUID="64e2e578-6bf9-4bc9-91ff-b93b154ee100" containerName="maas-api" containerID="cri-o://14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f" gracePeriod=30 Nov 04 01:19:37.946957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.946916 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2"] Nov 04 01:19:37.981410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.947968 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" containerName="payload-pre-processing" Nov 04 01:19:37.981410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.947996 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" containerName="payload-pre-processing" Nov 04 01:19:37.981410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.948013 4312 cpu_manager.go:414] "containerMap: removing container" podUID="47a0525c-e5c1-4562-aba3-644085829b40" containerName="payload-processing" Nov 04 01:19:37.981410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.948022 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a0525c-e5c1-4562-aba3-644085829b40" containerName="payload-processing" Nov 04 01:19:37.981410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.948256 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" containerName="payload-pre-processing" Nov 04 01:19:37.981410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.948275 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a0525c-e5c1-4562-aba3-644085829b40" containerName="payload-processing" Nov 04 01:19:37.992946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.992563 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2"] Nov 04 01:19:37.992946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:37.992699 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.004734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.004384 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-3bf4f9-dockercfg-cvkx2\"" Nov 04 01:19:38.007403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.006809 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-3bf4f9\"" Nov 04 01:19:38.110486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.109242 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b40aff9-f6df-482d-b40d-ec023527eb7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.110486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.109716 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjpr7\" (UniqueName: \"kubernetes.io/projected/3b40aff9-f6df-482d-b40d-ec023527eb7b-kube-api-access-bjpr7\") pod \"payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.365485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.348162 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjpr7\" (UniqueName: \"kubernetes.io/projected/3b40aff9-f6df-482d-b40d-ec023527eb7b-kube-api-access-bjpr7\") pod \"payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.365485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.348421 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b40aff9-f6df-482d-b40d-ec023527eb7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.365485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.349412 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b40aff9-f6df-482d-b40d-ec023527eb7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.578433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.563067 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjpr7\" (UniqueName: \"kubernetes.io/projected/3b40aff9-f6df-482d-b40d-ec023527eb7b-kube-api-access-bjpr7\") pod \"payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.638880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.637567 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:38.884408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.880021 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a0525c-e5c1-4562-aba3-644085829b40" path="/var/lib/kubelet/pods/47a0525c-e5c1-4562-aba3-644085829b40/volumes" Nov 04 01:19:38.884408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.884291 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03" path="/var/lib/kubelet/pods/a3c1ad7d-3ed1-4b2a-b76c-8dde8b7bcf03/volumes" Nov 04 01:19:38.904968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.898011 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n"] Nov 04 01:19:38.907161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.907124 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" podUID="fe166b36-1e15-448c-a77e-fb490258f463" containerName="istio-proxy" containerID="cri-o://738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35" gracePeriod=30 Nov 04 01:19:38.907713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.901939 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:38.912318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.911782 4312 generic.go:356] "Generic (PLEG): container finished" podID="64e2e578-6bf9-4bc9-91ff-b93b154ee100" containerID="14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f" exitCode=0 Nov 04 01:19:38.913574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.911809 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" event={"ID":"64e2e578-6bf9-4bc9-91ff-b93b154ee100","Type":"ContainerDied","Data":"14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f"} Nov 04 01:19:38.913748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.913728 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" event={"ID":"64e2e578-6bf9-4bc9-91ff-b93b154ee100","Type":"ContainerDied","Data":"93ccd653b0b008266aa2514d571d47177f2bd94b80cd660de9ecd96557b8ce29"} Nov 04 01:19:38.913863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:38.913850 4312 scope.go:122] "RemoveContainer" containerID="14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f" Nov 04 01:19:39.044540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.044381 4312 scope.go:122] "RemoveContainer" containerID="14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f" Nov 04 01:19:39.046398 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:39.046285 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f\": container with ID starting with 14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f not found: ID does not exist" containerID="14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f" Nov 04 01:19:39.046398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.046348 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f"} err="failed to get container status \"14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f\": rpc error: code = NotFound desc = could not find container \"14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f\": container with ID starting with 14e6404d835161ffdaa63606cc6c886ea18b3bfa3f62e4390f0c274dd0e7178f not found: ID does not exist" Nov 04 01:19:39.050059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.047901 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e2e578-6bf9-4bc9-91ff-b93b154ee100-kube-api-access-v4ltm\" (UniqueName: \"kubernetes.io/projected/64e2e578-6bf9-4bc9-91ff-b93b154ee100-kube-api-access-v4ltm\") pod \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " Nov 04 01:19:39.050059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.047977 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-maas-api-tls\") pod \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " Nov 04 01:19:39.050059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.048158 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-metrics-certs\") pod \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " Nov 04 01:19:39.050059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.048264 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e2e578-6bf9-4bc9-91ff-b93b154ee100-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e2e578-6bf9-4bc9-91ff-b93b154ee100-openshift-service-ca\") pod \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\" (UID: \"64e2e578-6bf9-4bc9-91ff-b93b154ee100\") " Nov 04 01:19:39.065720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.058229 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e2e578-6bf9-4bc9-91ff-b93b154ee100-openshift-service-ca" pod "64e2e578-6bf9-4bc9-91ff-b93b154ee100" (UID: "64e2e578-6bf9-4bc9-91ff-b93b154ee100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:39.088745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.084622 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e2e578-6bf9-4bc9-91ff-b93b154ee100-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:39.156428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.154503 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e2e578-6bf9-4bc9-91ff-b93b154ee100-kube-api-access-v4ltm" pod "64e2e578-6bf9-4bc9-91ff-b93b154ee100" (UID: "64e2e578-6bf9-4bc9-91ff-b93b154ee100"). InnerVolumeSpecName "kube-api-access-v4ltm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:39.309286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.308409 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded" Nov 04 01:19:39.365150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.349059 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:19:39.378216 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.376043 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4ltm\" (UniqueName: \"kubernetes.io/projected/64e2e578-6bf9-4bc9-91ff-b93b154ee100-kube-api-access-v4ltm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:39.480357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.480191 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-maas-api-tls" pod "64e2e578-6bf9-4bc9-91ff-b93b154ee100" (UID: "64e2e578-6bf9-4bc9-91ff-b93b154ee100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:39.480357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.480274 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-metrics-certs" pod "64e2e578-6bf9-4bc9-91ff-b93b154ee100" (UID: "64e2e578-6bf9-4bc9-91ff-b93b154ee100"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:39.502519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.502486 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:39.502740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.502723 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e2e578-6bf9-4bc9-91ff-b93b154ee100-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:39.627506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.627472 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2"] Nov 04 01:19:39.629087 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:39.629057 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b40aff9_f6df_482d_b40d_ec023527eb7b.slice/crio-4437dd1430bc97fd3eaee774f9ca66c8b2d2547d9c1a12affc0b86daac19a7d3 WatchSource:0}: Error finding container 4437dd1430bc97fd3eaee774f9ca66c8b2d2547d9c1a12affc0b86daac19a7d3: Status 404 returned error can't find the container with id 4437dd1430bc97fd3eaee774f9ca66c8b2d2547d9c1a12affc0b86daac19a7d3 Nov 04 01:19:39.974477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.974390 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj" Nov 04 01:19:39.994262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.987112 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" event={"ID":"3b40aff9-f6df-482d-b40d-ec023527eb7b","Type":"ContainerStarted","Data":"3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde"} Nov 04 01:19:39.994468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.994310 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:39.994468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:39.994372 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" event={"ID":"3b40aff9-f6df-482d-b40d-ec023527eb7b","Type":"ContainerStarted","Data":"4437dd1430bc97fd3eaee774f9ca66c8b2d2547d9c1a12affc0b86daac19a7d3"} Nov 04 01:19:40.044422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:40.043870 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" podStartSLOduration=3.04384686 podStartE2EDuration="3.04384686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:40.018741027 +0000 UTC m=+1012.041673030" watchObservedRunningTime="2026-11-04 01:19:40.04384686 +0000 UTC m=+1012.066778867" Nov 04 01:19:40.309538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:40.309497 4312 patch_prober.go:29] interesting pod/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:19:40.309768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:40.309741 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" podUID="fe166b36-1e15-448c-a77e-fb490258f463" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:19:40.316409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:40.316379 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj"] Nov 04 01:19:40.327112 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:40.327064 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cca527b7-5f747f7b4-92dlj"] Nov 04 01:19:42.116507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:42.116466 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e2e578-6bf9-4bc9-91ff-b93b154ee100" path="/var/lib/kubelet/pods/64e2e578-6bf9-4bc9-91ff-b93b154ee100/volumes" Nov 04 01:19:43.522584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.522546 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq"] Nov 04 01:19:43.524161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.524129 4312 cpu_manager.go:414] "containerMap: removing container" podUID="64e2e578-6bf9-4bc9-91ff-b93b154ee100" containerName="maas-api" Nov 04 01:19:43.524161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.524160 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e2e578-6bf9-4bc9-91ff-b93b154ee100" containerName="maas-api" Nov 04 01:19:43.524376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.524350 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e2e578-6bf9-4bc9-91ff-b93b154ee100" containerName="maas-api" Nov 04 01:19:43.530787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.530752 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.536815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.536789 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq"] Nov 04 01:19:43.545817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.545793 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-3bf4f9-serving-cert\"" Nov 04 01:19:43.579319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.579288 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lb64\" (UniqueName: \"kubernetes.io/projected/15f1368b-899a-4ca7-9edd-57b77f4657b5-kube-api-access-4lb64\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.579547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.579522 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f1368b-899a-4ca7-9edd-57b77f4657b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.579777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.579756 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.579853 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.579836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.681771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.681731 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lb64\" (UniqueName: \"kubernetes.io/projected/15f1368b-899a-4ca7-9edd-57b77f4657b5-kube-api-access-4lb64\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.681969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.681910 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f1368b-899a-4ca7-9edd-57b77f4657b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.682128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.682110 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.682198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.682183 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.683002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.682973 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f1368b-899a-4ca7-9edd-57b77f4657b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.685141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.685104 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.685320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.685277 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.758364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.756072 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lb64\" (UniqueName: \"kubernetes.io/projected/15f1368b-899a-4ca7-9edd-57b77f4657b5-kube-api-access-4lb64\") pod \"maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:43.872359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:43.864899 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:44.274434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:44.272114 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq"] Nov 04 01:19:45.103222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:45.103191 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" event={"ID":"15f1368b-899a-4ca7-9edd-57b77f4657b5","Type":"ContainerStarted","Data":"792eb790ee369a879c2f89b87950527169718267c67baf2696f7068b3cfa4eda"} Nov 04 01:19:45.788494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:45.788461 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4"] Nov 04 01:19:45.791565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:45.791537 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx"] Nov 04 01:19:46.354025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:46.338487 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" event={"ID":"15f1368b-899a-4ca7-9edd-57b77f4657b5","Type":"ContainerStarted","Data":"5157ddb64065f50d2310ab044458bff8ab5da696b1854f23880f5eba2f2508b0"} Nov 04 01:19:46.364354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:46.355658 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:46.422366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:46.411712 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" podStartSLOduration=2.89876013 podStartE2EDuration="3.411692354s" totalImagesPullingTime="512.932224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:46.396546357 +0000 UTC m=+1018.419478387" watchObservedRunningTime="2026-11-04 01:19:46.411692354 +0000 UTC m=+1018.434624385" Nov 04 01:19:48.727411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.727352 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:19:48.824761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.824722 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-kube-api-access-zmcmk\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-kube-api-access-zmcmk\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.824954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.824793 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-istio-token\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-istio-token\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.824954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.824826 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-socket\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.824954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.824867 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-data\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.825131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.824988 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-certs\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.825192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825135 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-credential-socket\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.825192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825153 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-socket" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:48.825192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825174 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe166b36-1e15-448c-a77e-fb490258f463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166b36-1e15-448c-a77e-fb490258f463-istiod-ca-cert\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.825192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825162 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-data" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:48.825418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825244 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-envoy\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.825418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825357 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-certs" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:48.825418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825384 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-credential-socket" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:48.825657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825634 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe166b36-1e15-448c-a77e-fb490258f463-istiod-ca-cert" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:48.825743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.825679 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe166b36-1e15-448c-a77e-fb490258f463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166b36-1e15-448c-a77e-fb490258f463-istio-podinfo\") pod \"fe166b36-1e15-448c-a77e-fb490258f463\" (UID: \"fe166b36-1e15-448c-a77e-fb490258f463\") " Nov 04 01:19:48.826514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.826488 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.826632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.826518 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.826632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.826533 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.826632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.826548 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.826632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.826562 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166b36-1e15-448c-a77e-fb490258f463-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.828005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.827977 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-istio-token" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:48.828216 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.828198 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-envoy" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:48.828395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.828375 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe166b36-1e15-448c-a77e-fb490258f463-istio-podinfo" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:19:48.828479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.828402 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-kube-api-access-zmcmk" pod "fe166b36-1e15-448c-a77e-fb490258f463" (UID: "fe166b36-1e15-448c-a77e-fb490258f463"). InnerVolumeSpecName "kube-api-access-zmcmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:48.954398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.947855 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166b36-1e15-448c-a77e-fb490258f463-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.954398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.947894 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166b36-1e15-448c-a77e-fb490258f463-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.954398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.947911 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcmk\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-kube-api-access-zmcmk\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:48.954398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:48.947925 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166b36-1e15-448c-a77e-fb490258f463-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:49.431041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.430952 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe166b36-1e15-448c-a77e-fb490258f463" containerID="738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35" exitCode=0 Nov 04 01:19:49.431041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.431016 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" Nov 04 01:19:49.431261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.431032 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" event={"ID":"fe166b36-1e15-448c-a77e-fb490258f463","Type":"ContainerDied","Data":"738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35"} Nov 04 01:19:49.431261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.431114 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n" event={"ID":"fe166b36-1e15-448c-a77e-fb490258f463","Type":"ContainerDied","Data":"2c6a68f1a45059a9d7f1d4685032a7cdaf47ad9cf1706cb0e4922585c22e1b11"} Nov 04 01:19:49.431261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.431141 4312 scope.go:122] "RemoveContainer" containerID="738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35" Nov 04 01:19:49.442775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.442757 4312 scope.go:122] "RemoveContainer" containerID="738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35" Nov 04 01:19:49.443042 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:49.443023 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35\": container with ID starting with 738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35 not found: ID does not exist" containerID="738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35" Nov 04 01:19:49.443092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.443050 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35"} err="failed to get container status \"738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35\": rpc error: code = NotFound desc = could not find container \"738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35\": container with ID starting with 738a228806304f5a84b81f92e367039228267b005ca1ff67614bb8859a7a0a35 not found: ID does not exist" Nov 04 01:19:49.457178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.457155 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n"] Nov 04 01:19:49.461844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:49.461823 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cca527b7-gw-openshift-default-bffdd4797-zsj9n"] Nov 04 01:19:50.110404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:50.110374 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe166b36-1e15-448c-a77e-fb490258f463" path="/var/lib/kubelet/pods/fe166b36-1e15-448c-a77e-fb490258f463/volumes" Nov 04 01:19:50.790094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:50.790059 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" podUID="ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" containerName="payload-processing" containerID="cri-o://81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71" gracePeriod=25 Nov 04 01:19:50.792547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:50.792519 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" podUID="9adbfb87-1c2d-4c81-96bc-d31541bb6beb" containerName="payload-pre-processing" containerID="cri-o://e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8" gracePeriod=25 Nov 04 01:19:50.917693 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:50.917664 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae05a28f_933b_42b5_b698_ea5f5b0cdd7a.slice/crio-conmon-81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:19:51.132405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.129525 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:19:51.235073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.235049 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:51.244034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.244005 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:51.414432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.414350 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-plugins-config-volume\") pod \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " Nov 04 01:19:51.414591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.414500 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-kube-api-access-9g9lb\" (UniqueName: \"kubernetes.io/projected/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-kube-api-access-9g9lb\") pod \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " Nov 04 01:19:51.414591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.414543 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-plugins-config-volume\") pod \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\" (UID: \"9adbfb87-1c2d-4c81-96bc-d31541bb6beb\") " Nov 04 01:19:51.414723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.414636 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-kube-api-access-ttjtt\" (UniqueName: \"kubernetes.io/projected/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-kube-api-access-ttjtt\") pod \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\" (UID: \"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a\") " Nov 04 01:19:51.414723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.414689 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-plugins-config-volume" pod "ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" (UID: "ae05a28f-933b-42b5-b698-ea5f5b0cdd7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:51.415148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.415119 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-plugins-config-volume" pod "9adbfb87-1c2d-4c81-96bc-d31541bb6beb" (UID: "9adbfb87-1c2d-4c81-96bc-d31541bb6beb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:51.415407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.415384 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:51.415516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.415411 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:51.416950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.416922 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-kube-api-access-ttjtt" pod "ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" (UID: "ae05a28f-933b-42b5-b698-ea5f5b0cdd7a"). InnerVolumeSpecName "kube-api-access-ttjtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:51.416950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.416943 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-kube-api-access-9g9lb" pod "9adbfb87-1c2d-4c81-96bc-d31541bb6beb" (UID: "9adbfb87-1c2d-4c81-96bc-d31541bb6beb"). InnerVolumeSpecName "kube-api-access-9g9lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:51.448595 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.448569 4312 generic.go:356] "Generic (PLEG): container finished" podID="9adbfb87-1c2d-4c81-96bc-d31541bb6beb" containerID="e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8" exitCode=0 Nov 04 01:19:51.448708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.448614 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" Nov 04 01:19:51.448708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.448660 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" event={"ID":"9adbfb87-1c2d-4c81-96bc-d31541bb6beb","Type":"ContainerDied","Data":"e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8"} Nov 04 01:19:51.448833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.448718 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx" event={"ID":"9adbfb87-1c2d-4c81-96bc-d31541bb6beb","Type":"ContainerDied","Data":"b1732900c96816a35ec5d1af71c979c1cadd4d629f112b80208ef1fb1f067856"} Nov 04 01:19:51.448833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.448750 4312 scope.go:122] "RemoveContainer" containerID="e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8" Nov 04 01:19:51.454399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.454373 4312 generic.go:356] "Generic (PLEG): container finished" podID="ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" containerID="81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71" exitCode=0 Nov 04 01:19:51.454519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.454411 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" event={"ID":"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a","Type":"ContainerDied","Data":"81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71"} Nov 04 01:19:51.454519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.454437 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" Nov 04 01:19:51.454519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.454455 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4" event={"ID":"ae05a28f-933b-42b5-b698-ea5f5b0cdd7a","Type":"ContainerDied","Data":"0f6593cd64e7f202c008388b384b1cbcc43eabca33657445d41130c31f968485"} Nov 04 01:19:51.461886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.461870 4312 scope.go:122] "RemoveContainer" containerID="e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8" Nov 04 01:19:51.462194 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:51.462169 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8\": container with ID starting with e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8 not found: ID does not exist" containerID="e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8" Nov 04 01:19:51.462250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.462209 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8"} err="failed to get container status \"e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8\": rpc error: code = NotFound desc = could not find container \"e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8\": container with ID starting with e7306b78bcc7a19c23963a286d769384d39b753c99ddc056221bcf3ca6321de8 not found: ID does not exist" Nov 04 01:19:51.462250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.462238 4312 scope.go:122] "RemoveContainer" containerID="81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71" Nov 04 01:19:51.471728 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.471711 4312 scope.go:122] "RemoveContainer" containerID="81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71" Nov 04 01:19:51.472039 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:51.472002 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71\": container with ID starting with 81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71 not found: ID does not exist" containerID="81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71" Nov 04 01:19:51.472241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.472087 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71"} err="failed to get container status \"81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71\": rpc error: code = NotFound desc = could not find container \"81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71\": container with ID starting with 81ddd99daa8719416efec670622d5a3183a4809c2dc69d45a217e5452b37ad71 not found: ID does not exist" Nov 04 01:19:51.477274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.477254 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx"] Nov 04 01:19:51.487051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.487022 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ab7c587-5d6df64cd7-sbnnx"] Nov 04 01:19:51.498285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.498235 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4"] Nov 04 01:19:51.502365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.502347 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ab7c587-64544bc694-62kt4"] Nov 04 01:19:51.517568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.517544 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9lb\" (UniqueName: \"kubernetes.io/projected/9adbfb87-1c2d-4c81-96bc-d31541bb6beb-kube-api-access-9g9lb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:51.517568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.517567 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttjtt\" (UniqueName: \"kubernetes.io/projected/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a-kube-api-access-ttjtt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:51.607288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.607239 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk"] Nov 04 01:19:51.607582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:51.607532 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" podUID="b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" containerName="maas-api" containerID="cri-o://56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed" gracePeriod=30 Nov 04 01:19:52.402917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.402803 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9adbfb87-1c2d-4c81-96bc-d31541bb6beb" path="/var/lib/kubelet/pods/9adbfb87-1c2d-4c81-96bc-d31541bb6beb/volumes" Nov 04 01:19:52.410440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.410410 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" path="/var/lib/kubelet/pods/ae05a28f-933b-42b5-b698-ea5f5b0cdd7a/volumes" Nov 04 01:19:52.411084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.411056 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r"] Nov 04 01:19:52.411445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.411419 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" podUID="05d0b0f1-63aa-4d98-a86b-e19c310cf369" containerName="istio-proxy" containerID="cri-o://cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451" gracePeriod=30 Nov 04 01:19:52.531357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.511968 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:52.537020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.536996 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" Nov 04 01:19:52.537882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.537325 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" event={"ID":"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d","Type":"ContainerDied","Data":"56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed"} Nov 04 01:19:52.538095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.538081 4312 scope.go:122] "RemoveContainer" containerID="56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed" Nov 04 01:19:52.538290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.536881 4312 generic.go:356] "Generic (PLEG): container finished" podID="b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" containerID="56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed" exitCode=0 Nov 04 01:19:52.589373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.563405 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk" event={"ID":"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d","Type":"ContainerDied","Data":"18a316b422292d0e884b212473e2908c11bffaa6c7588b5b4aa2551ce9d367ad"} Nov 04 01:19:52.627943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.614058 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-metrics-certs\") pod \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " Nov 04 01:19:52.627943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.614135 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-maas-api-tls\") pod \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " Nov 04 01:19:52.627943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.614259 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-openshift-service-ca\") pod \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " Nov 04 01:19:52.627943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.614488 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-kube-api-access-fg6t5\" (UniqueName: \"kubernetes.io/projected/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-kube-api-access-fg6t5\") pod \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\" (UID: \"b9949f51-5fe1-44d1-b760-0a5a1bb0a18d\") " Nov 04 01:19:52.649773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.649712 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-openshift-service-ca" pod "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" (UID: "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:52.650448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.650416 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-maas-api-tls" pod "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" (UID: "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:52.665710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.663534 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-kube-api-access-fg6t5" pod "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" (UID: "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d"). InnerVolumeSpecName "kube-api-access-fg6t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:52.672381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.672064 4312 scope.go:122] "RemoveContainer" containerID="56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed" Nov 04 01:19:52.672863 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:52.672762 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed\": container with ID starting with 56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed not found: ID does not exist" containerID="56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed" Nov 04 01:19:52.672863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.672798 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed"} err="failed to get container status \"56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed\": rpc error: code = NotFound desc = could not find container \"56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed\": container with ID starting with 56c7b7056237f36e2e0f4e4ded60b5af81ed01f1c657fbf89e36e4fe8e2037ed not found: ID does not exist" Nov 04 01:19:52.675529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.675273 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-metrics-certs" pod "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" (UID: "b9949f51-5fe1-44d1-b760-0a5a1bb0a18d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:52.728357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.725772 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg6t5\" (UniqueName: \"kubernetes.io/projected/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-kube-api-access-fg6t5\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:52.728357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.725812 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:52.728357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.725829 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:52.728357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.725844 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:52.864133 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.864103 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk"] Nov 04 01:19:52.873723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:52.873696 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ab7c587-c544bf6b7-mlbjk"] Nov 04 01:19:53.426877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.426839 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:19:53.491250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.491217 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf"] Nov 04 01:19:53.492084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492065 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe166b36-1e15-448c-a77e-fb490258f463" containerName="istio-proxy" Nov 04 01:19:53.492136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492086 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe166b36-1e15-448c-a77e-fb490258f463" containerName="istio-proxy" Nov 04 01:19:53.492136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492103 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" containerName="maas-api" Nov 04 01:19:53.492136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492109 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" containerName="maas-api" Nov 04 01:19:53.492136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492119 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9adbfb87-1c2d-4c81-96bc-d31541bb6beb" containerName="payload-pre-processing" Nov 04 01:19:53.492136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492125 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9adbfb87-1c2d-4c81-96bc-d31541bb6beb" containerName="payload-pre-processing" Nov 04 01:19:53.492296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492164 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" containerName="payload-processing" Nov 04 01:19:53.492296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492170 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" containerName="payload-processing" Nov 04 01:19:53.492296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492266 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae05a28f-933b-42b5-b698-ea5f5b0cdd7a" containerName="payload-processing" Nov 04 01:19:53.492296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492276 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe166b36-1e15-448c-a77e-fb490258f463" containerName="istio-proxy" Nov 04 01:19:53.492296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492284 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" containerName="maas-api" Nov 04 01:19:53.492296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.492291 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9adbfb87-1c2d-4c81-96bc-d31541bb6beb" containerName="payload-pre-processing" Nov 04 01:19:53.495618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.495601 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.514500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.514477 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-23ef8f-openshift-default-dockercfg-mkx7c\"" Nov 04 01:19:53.555749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.555708 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istiod-ca-cert\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.555921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.555889 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-credential-socket\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.556048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.556029 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-certs\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.556092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.556077 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-envoy\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.556294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.556271 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-token\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.556384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.556365 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-data\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.556426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.556410 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmq6d\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-kube-api-access-dmq6d\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.556460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.556441 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-podinfo\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.556568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.556556 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-socket\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.564312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.564286 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf"] Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.694541 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-socket\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.694687 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istiod-ca-cert\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.694797 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-credential-socket\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.694896 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-certs\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.694945 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-envoy\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.695052 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-token\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.695104 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-data\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.695152 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmq6d\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-kube-api-access-dmq6d\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.700409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.695195 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-podinfo\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.744364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.719782 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-certs\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.744364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.720061 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-socket\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.744364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.720878 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istiod-ca-cert\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.744364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.721112 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-credential-socket\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.744364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.729656 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-data\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.744364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.736997 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-token\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.770359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.767551 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-podinfo\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.834965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.834084 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-envoy\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:53.845575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:53.840595 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmq6d\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-kube-api-access-dmq6d\") pod \"e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:54.111471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.111440 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9949f51-5fe1-44d1-b760-0a5a1bb0a18d" path="/var/lib/kubelet/pods/b9949f51-5fe1-44d1-b760-0a5a1bb0a18d/volumes" Nov 04 01:19:54.112127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.112105 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:54.414980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.414884 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf"] Nov 04 01:19:54.422809 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:54.422770 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d79be2_e8ca_417a_bd4b_aa14b8a237fa.slice/crio-5e18d172a91fdbb9a74928d7f14d29b54cfa6c064a9d174ad063753ac4421e89 WatchSource:0}: Error finding container 5e18d172a91fdbb9a74928d7f14d29b54cfa6c064a9d174ad063753ac4421e89: Status 404 returned error can't find the container with id 5e18d172a91fdbb9a74928d7f14d29b54cfa6c064a9d174ad063753ac4421e89 Nov 04 01:19:54.425773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.425738 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:54.425880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.425848 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:54.425944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.425895 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:19:54.937315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.937276 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" event={"ID":"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa","Type":"ContainerStarted","Data":"10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a"} Nov 04 01:19:54.937722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.937374 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" event={"ID":"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa","Type":"ContainerStarted","Data":"5e18d172a91fdbb9a74928d7f14d29b54cfa6c064a9d174ad063753ac4421e89"} Nov 04 01:19:54.970022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:54.969976 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podStartSLOduration=1.9699622209999998 podStartE2EDuration="1.969962221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:54.965757287 +0000 UTC m=+1026.988689312" watchObservedRunningTime="2026-11-04 01:19:54.969962221 +0000 UTC m=+1026.992894228" Nov 04 01:19:55.136355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:55.128419 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:55.165369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:55.165315 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:55.945040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:55.945012 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:55.945983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:55.945956 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:19:55.947137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:55.947117 4312 patch_prober.go:29] interesting pod/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 04 01:19:55.947205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:55.947164 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" podUID="05d0b0f1-63aa-4d98-a86b-e19c310cf369" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:19:57.744536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.744471 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:57.745681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.745662 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-token\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-token\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.745777 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.745762 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istiod-ca-cert\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.745849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.745788 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-envoy\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.745849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.745824 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-credential-socket\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.745950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.745867 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-socket\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.746041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.746025 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-certs\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.746210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.746166 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-credential-socket" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:57.746210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.746180 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istiod-ca-cert" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:57.746210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.746203 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-socket" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:57.746467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.746385 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-certs" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:57.746986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.746963 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.747035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.746995 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.747035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.747011 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.747035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.747026 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.747843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.747820 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-envoy" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:57.748095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.748072 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-token" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:57.848530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.848495 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-kube-api-access-z579r\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-kube-api-access-z579r\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.848696 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.848635 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-data\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.848696 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.848687 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-podinfo\") pod \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\" (UID: \"05d0b0f1-63aa-4d98-a86b-e19c310cf369\") " Nov 04 01:19:57.848928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.848903 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-data" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:19:57.849437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.849420 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.849504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.849437 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.849504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.849451 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.850669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.850647 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-kube-api-access-z579r" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "kube-api-access-z579r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:57.850765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.850648 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-podinfo" pod "05d0b0f1-63aa-4d98-a86b-e19c310cf369" (UID: "05d0b0f1-63aa-4d98-a86b-e19c310cf369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:19:57.951241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.951201 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d0b0f1-63aa-4d98-a86b-e19c310cf369-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.951241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.951226 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z579r\" (UniqueName: \"kubernetes.io/projected/05d0b0f1-63aa-4d98-a86b-e19c310cf369-kube-api-access-z579r\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:57.960780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.960753 4312 generic.go:356] "Generic (PLEG): container finished" podID="05d0b0f1-63aa-4d98-a86b-e19c310cf369" containerID="cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451" exitCode=0 Nov 04 01:19:57.960915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.960805 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" Nov 04 01:19:57.960915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.960833 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" event={"ID":"05d0b0f1-63aa-4d98-a86b-e19c310cf369","Type":"ContainerDied","Data":"cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451"} Nov 04 01:19:57.960915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.960884 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r" event={"ID":"05d0b0f1-63aa-4d98-a86b-e19c310cf369","Type":"ContainerDied","Data":"72d55d434980416846304d3ea1c3f8c3a13ba78093a885ba11c8ac9afa400833"} Nov 04 01:19:57.960915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.960912 4312 scope.go:122] "RemoveContainer" containerID="cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451" Nov 04 01:19:57.973328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.973306 4312 scope.go:122] "RemoveContainer" containerID="cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451" Nov 04 01:19:57.973633 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:57.973611 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451\": container with ID starting with cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451 not found: ID does not exist" containerID="cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451" Nov 04 01:19:57.973699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.973641 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451"} err="failed to get container status \"cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451\": rpc error: code = NotFound desc = could not find container \"cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451\": container with ID starting with cbd3e3b9b83f58573fd2e9c41a4b809e825cb76eb1d4fd699a918921512c0451 not found: ID does not exist" Nov 04 01:19:57.995316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.995241 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r"] Nov 04 01:19:57.998710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:57.998685 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8ab7c587-openshift-default-6796fdc5bf-7wh4r"] Nov 04 01:19:58.156203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.156173 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d0b0f1-63aa-4d98-a86b-e19c310cf369" path="/var/lib/kubelet/pods/05d0b0f1-63aa-4d98-a86b-e19c310cf369/volumes" Nov 04 01:19:58.532136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.532102 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj"] Nov 04 01:19:58.532950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.532927 4312 cpu_manager.go:414] "containerMap: removing container" podUID="05d0b0f1-63aa-4d98-a86b-e19c310cf369" containerName="istio-proxy" Nov 04 01:19:58.532950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.532949 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d0b0f1-63aa-4d98-a86b-e19c310cf369" containerName="istio-proxy" Nov 04 01:19:58.533141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.533128 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d0b0f1-63aa-4d98-a86b-e19c310cf369" containerName="istio-proxy" Nov 04 01:19:58.536601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.536577 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.541263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.541238 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx"] Nov 04 01:19:58.544852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.544817 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-23ef8f\"" Nov 04 01:19:58.545159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.544995 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-23ef8f-dockercfg-29pgb\"" Nov 04 01:19:58.545462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.545200 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:58.548146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.548126 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj"] Nov 04 01:19:58.564650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.564620 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx"] Nov 04 01:19:58.576065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.576036 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhrgp\" (UniqueName: \"kubernetes.io/projected/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-kube-api-access-xhrgp\") pod \"payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:58.576197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.576111 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.576261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.576197 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qccc\" (UniqueName: \"kubernetes.io/projected/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-kube-api-access-7qccc\") pod \"payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.576577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.576554 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:58.699418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.692094 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:58.699418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.692411 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhrgp\" (UniqueName: \"kubernetes.io/projected/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-kube-api-access-xhrgp\") pod \"payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:58.699418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.692538 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.699418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.692624 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qccc\" (UniqueName: \"kubernetes.io/projected/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-kube-api-access-7qccc\") pod \"payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.699418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.694385 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:58.703439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.701722 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.729392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.726117 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qccc\" (UniqueName: \"kubernetes.io/projected/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-kube-api-access-7qccc\") pod \"payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.736376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.730231 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhrgp\" (UniqueName: \"kubernetes.io/projected/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-kube-api-access-xhrgp\") pod \"payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:58.857496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.856644 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:19:58.886589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:58.884692 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:19:59.463876 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:19:59.463846 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:19:59.466808 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:59.466778 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20df1fd2_d5ca_4a9a_8280_fed7277e18ac.slice/crio-30d9bbf3d82b08d6d3ce924fa2c9636c67de7866d80d721c15c55c97b77b2993 WatchSource:0}: Error finding container 30d9bbf3d82b08d6d3ce924fa2c9636c67de7866d80d721c15c55c97b77b2993: Status 404 returned error can't find the container with id 30d9bbf3d82b08d6d3ce924fa2c9636c67de7866d80d721c15c55c97b77b2993 Nov 04 01:19:59.467004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:59.466985 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx"] Nov 04 01:19:59.569071 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:19:59.569043 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc1c7f0_aa3e_4dd9_a58c_14795e9411b6.slice/crio-cf2ff512ef75b9e76016ed87683e29d359a69bbdb329cbc4cfceb9cf576bf679 WatchSource:0}: Error finding container cf2ff512ef75b9e76016ed87683e29d359a69bbdb329cbc4cfceb9cf576bf679: Status 404 returned error can't find the container with id cf2ff512ef75b9e76016ed87683e29d359a69bbdb329cbc4cfceb9cf576bf679 Nov 04 01:19:59.569412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:19:59.569388 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj"] Nov 04 01:20:00.256244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.256211 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" event={"ID":"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6","Type":"ContainerStarted","Data":"ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8"} Nov 04 01:20:00.256740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.256268 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" event={"ID":"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6","Type":"ContainerStarted","Data":"cf2ff512ef75b9e76016ed87683e29d359a69bbdb329cbc4cfceb9cf576bf679"} Nov 04 01:20:00.256740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.256297 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:20:00.257940 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.257916 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" event={"ID":"20df1fd2-d5ca-4a9a-8280-fed7277e18ac","Type":"ContainerStarted","Data":"8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915"} Nov 04 01:20:00.258046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.257957 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" event={"ID":"20df1fd2-d5ca-4a9a-8280-fed7277e18ac","Type":"ContainerStarted","Data":"30d9bbf3d82b08d6d3ce924fa2c9636c67de7866d80d721c15c55c97b77b2993"} Nov 04 01:20:00.258046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.257984 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:20:00.276242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.276190 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" podStartSLOduration=2.276176603 podStartE2EDuration="2.276176603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:00.274742637 +0000 UTC m=+1032.297674644" watchObservedRunningTime="2026-11-04 01:20:00.276176603 +0000 UTC m=+1032.299108656" Nov 04 01:20:00.297604 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:00.297554 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" podStartSLOduration=2.2975422 podStartE2EDuration="2.2975422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:00.293401608 +0000 UTC m=+1032.316333644" watchObservedRunningTime="2026-11-04 01:20:00.2975422 +0000 UTC m=+1032.320474226" Nov 04 01:20:04.626622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.617667 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7"] Nov 04 01:20:04.636773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.636745 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.640773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.640744 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-23ef8f-serving-cert\"" Nov 04 01:20:04.642473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.642444 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7"] Nov 04 01:20:04.795408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.794721 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwz7n\" (UniqueName: \"kubernetes.io/projected/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-kube-api-access-vwz7n\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.795408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.794861 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.795408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.794921 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.795408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.795144 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.897108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.897019 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.897108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.897075 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.897367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.897217 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.897367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.897354 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwz7n\" (UniqueName: \"kubernetes.io/projected/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-kube-api-access-vwz7n\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.898022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.897992 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.899836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.899808 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.899944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.899923 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.909837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.909816 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwz7n\" (UniqueName: \"kubernetes.io/projected/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-kube-api-access-vwz7n\") pod \"maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:04.952678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:04.952648 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:05.030039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:05.029969 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z"] Nov 04 01:20:05.037484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:05.037451 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a657aa-gdv9z"] Nov 04 01:20:05.237199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:05.237157 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7"] Nov 04 01:20:05.237352 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:20:05.237251 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9beada2e_b29f_4762_8e9d_fd99c4a4fe0c.slice/crio-d7359a3caa320319bc5d2386b9c5bd0f85eca2a6648d1b2ef33f61f2b8d4aefc WatchSource:0}: Error finding container d7359a3caa320319bc5d2386b9c5bd0f85eca2a6648d1b2ef33f61f2b8d4aefc: Status 404 returned error can't find the container with id d7359a3caa320319bc5d2386b9c5bd0f85eca2a6648d1b2ef33f61f2b8d4aefc Nov 04 01:20:05.238957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:05.238938 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:20:05.374367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:05.373305 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" event={"ID":"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c","Type":"ContainerStarted","Data":"d7359a3caa320319bc5d2386b9c5bd0f85eca2a6648d1b2ef33f61f2b8d4aefc"} Nov 04 01:20:06.170700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:06.170665 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3fd24d-a34a-4d8e-b315-0870d6ae29f3" path="/var/lib/kubelet/pods/3f3fd24d-a34a-4d8e-b315-0870d6ae29f3/volumes" Nov 04 01:20:06.384197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:06.384100 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" event={"ID":"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c","Type":"ContainerStarted","Data":"44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1"} Nov 04 01:20:06.384197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:06.384168 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:06.408199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:06.408142 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" podStartSLOduration=1.818571886 podStartE2EDuration="2.408124975s" totalImagesPullingTime="589.553089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:06.403288273 +0000 UTC m=+1038.426220281" watchObservedRunningTime="2026-11-04 01:20:06.408124975 +0000 UTC m=+1038.431056988" Nov 04 01:20:07.031518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:07.031490 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64"] Nov 04 01:20:07.038924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:07.038896 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a464071e-ktt64"] Nov 04 01:20:08.320359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:08.316804 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd44ceb-72d0-4d89-899c-7ef158d1a9a5" path="/var/lib/kubelet/pods/1bd44ceb-72d0-4d89-899c-7ef158d1a9a5/volumes" Nov 04 01:20:11.268779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:11.268751 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:20:11.269184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:11.269163 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:20:12.454814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:12.454785 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:20:15.308055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.308022 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm"] Nov 04 01:20:15.314101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.314082 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.319688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.319644 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e7742169\"" Nov 04 01:20:15.320004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.319874 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e7742169-dockercfg-czvxd\"" Nov 04 01:20:15.323773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.323535 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm"] Nov 04 01:20:15.337903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.337873 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa96b9c-c094-4d25-b5e0-de31751312e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.338113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.338091 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4x82\" (UniqueName: \"kubernetes.io/projected/faa96b9c-c094-4d25-b5e0-de31751312e3-kube-api-access-m4x82\") pod \"payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.376062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.376022 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94"] Nov 04 01:20:15.414865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.414824 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94"] Nov 04 01:20:15.415063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.414952 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:15.475362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.474758 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa96b9c-c094-4d25-b5e0-de31751312e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.475362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.474931 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4x82\" (UniqueName: \"kubernetes.io/projected/faa96b9c-c094-4d25-b5e0-de31751312e3-kube-api-access-m4x82\") pod \"payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.475362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.475038 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b880e25-057e-4833-9dcb-a2c529abeb64-plugins-config-volume\") pod \"payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:15.488040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.488005 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa96b9c-c094-4d25-b5e0-de31751312e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.493528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.493490 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dffvn\" (UniqueName: \"kubernetes.io/projected/1b880e25-057e-4833-9dcb-a2c529abeb64-kube-api-access-dffvn\") pod \"payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:15.577201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.575408 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4x82\" (UniqueName: \"kubernetes.io/projected/faa96b9c-c094-4d25-b5e0-de31751312e3-kube-api-access-m4x82\") pod \"payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.601860 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.600634 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dffvn\" (UniqueName: \"kubernetes.io/projected/1b880e25-057e-4833-9dcb-a2c529abeb64-kube-api-access-dffvn\") pod \"payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:15.601860 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.600932 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b880e25-057e-4833-9dcb-a2c529abeb64-plugins-config-volume\") pod \"payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:15.638355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.603188 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b880e25-057e-4833-9dcb-a2c529abeb64-plugins-config-volume\") pod \"payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:15.689122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.648449 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dffvn\" (UniqueName: \"kubernetes.io/projected/1b880e25-057e-4833-9dcb-a2c529abeb64-kube-api-access-dffvn\") pod \"payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:15.693417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.659146 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:15.988766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:15.988737 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:16.026297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.026262 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr"] Nov 04 01:20:16.030768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.030739 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc288452-9wcqr"] Nov 04 01:20:16.116536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.116502 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a33c19-2e92-4037-b663-90074e2ea9f0" path="/var/lib/kubelet/pods/c5a33c19-2e92-4037-b663-90074e2ea9f0/volumes" Nov 04 01:20:16.175573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.175542 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm"] Nov 04 01:20:16.175690 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:20:16.175584 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa96b9c_c094_4d25_b5e0_de31751312e3.slice/crio-1aea6450e13943c7d78c081b838d78e82d24e6f0e56a09023c2b41a1e5f7d4a4 WatchSource:0}: Error finding container 1aea6450e13943c7d78c081b838d78e82d24e6f0e56a09023c2b41a1e5f7d4a4: Status 404 returned error can't find the container with id 1aea6450e13943c7d78c081b838d78e82d24e6f0e56a09023c2b41a1e5f7d4a4 Nov 04 01:20:16.378432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.371763 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94"] Nov 04 01:20:16.387364 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:20:16.386385 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b880e25_057e_4833_9dcb_a2c529abeb64.slice/crio-66dfa6c5df118a56f7fd1f4be0645b32ba853448f0d8db26bd2f318e7794efc0 WatchSource:0}: Error finding container 66dfa6c5df118a56f7fd1f4be0645b32ba853448f0d8db26bd2f318e7794efc0: Status 404 returned error can't find the container with id 66dfa6c5df118a56f7fd1f4be0645b32ba853448f0d8db26bd2f318e7794efc0 Nov 04 01:20:16.916039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.916002 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" event={"ID":"1b880e25-057e-4833-9dcb-a2c529abeb64","Type":"ContainerStarted","Data":"a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba"} Nov 04 01:20:16.916264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.916068 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" event={"ID":"1b880e25-057e-4833-9dcb-a2c529abeb64","Type":"ContainerStarted","Data":"66dfa6c5df118a56f7fd1f4be0645b32ba853448f0d8db26bd2f318e7794efc0"} Nov 04 01:20:16.916264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.916201 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:16.917750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.917722 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" event={"ID":"faa96b9c-c094-4d25-b5e0-de31751312e3","Type":"ContainerStarted","Data":"faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456"} Nov 04 01:20:16.917869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.917773 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" event={"ID":"faa96b9c-c094-4d25-b5e0-de31751312e3","Type":"ContainerStarted","Data":"1aea6450e13943c7d78c081b838d78e82d24e6f0e56a09023c2b41a1e5f7d4a4"} Nov 04 01:20:16.917869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:16.917819 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:17.068385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:17.065470 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" podStartSLOduration=2.06544816 podStartE2EDuration="2.06544816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:16.985079114 +0000 UTC m=+1049.008011141" watchObservedRunningTime="2026-11-04 01:20:17.06544816 +0000 UTC m=+1049.088380169" Nov 04 01:20:17.159365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:17.153369 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" podStartSLOduration=2.153326699 podStartE2EDuration="2.153326699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:17.022027186 +0000 UTC m=+1049.044959218" watchObservedRunningTime="2026-11-04 01:20:17.153326699 +0000 UTC m=+1049.176258707" Nov 04 01:20:22.558138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.558096 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7"] Nov 04 01:20:22.564383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.564360 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.571703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.571680 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e7742169-serving-cert\"" Nov 04 01:20:22.577765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.577737 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7"] Nov 04 01:20:22.662312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.662278 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae3bde8-b116-422a-92a7-2e23f76a71b2-openshift-service-ca\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.662555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.662384 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csxs2\" (UniqueName: \"kubernetes.io/projected/1ae3bde8-b116-422a-92a7-2e23f76a71b2-kube-api-access-csxs2\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.662555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.662551 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-metrics-certs\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.662652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.662581 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-maas-api-tls\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.791510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.791423 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-maas-api-tls\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.792118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.792084 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae3bde8-b116-422a-92a7-2e23f76a71b2-openshift-service-ca\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.813681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.813530 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csxs2\" (UniqueName: \"kubernetes.io/projected/1ae3bde8-b116-422a-92a7-2e23f76a71b2-kube-api-access-csxs2\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.814734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.814704 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-metrics-certs\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.829810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.796255 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae3bde8-b116-422a-92a7-2e23f76a71b2-openshift-service-ca\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.852356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.836255 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-metrics-certs\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.852356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.842978 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-maas-api-tls\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.853566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.853535 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csxs2\" (UniqueName: \"kubernetes.io/projected/1ae3bde8-b116-422a-92a7-2e23f76a71b2-kube-api-access-csxs2\") pod \"maas-api-e2e-mt-e7742169-6d555879d7-jlpd7\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:22.892318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:22.892285 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:23.387368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.384167 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7"] Nov 04 01:20:23.579915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.579840 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx"] Nov 04 01:20:23.586127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.586103 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.609959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.609930 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx"] Nov 04 01:20:23.613242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.613200 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-35980d-openshift-default-dockercfg-r4w7j\"" Nov 04 01:20:23.665736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.665703 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-data\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.665891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.665827 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-token\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.665891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.665882 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-credential-socket\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.666037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.666017 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-podinfo\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.666136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.666122 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-envoy\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.666186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.666145 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdsb\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-kube-api-access-vhdsb\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.666186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.666164 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-socket\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.666299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.666195 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-certs\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.666397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.666289 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5339c3-a472-44fe-8300-83fdf9114bc5-istiod-ca-cert\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.767761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.767726 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-token\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.767936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.767795 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-credential-socket\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.767936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.767852 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-podinfo\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.767936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.767907 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-envoy\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.767936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.767924 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdsb\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-kube-api-access-vhdsb\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768107 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-socket\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768204 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-certs\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768251 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5339c3-a472-44fe-8300-83fdf9114bc5-istiod-ca-cert\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768459 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-socket\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768528 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-certs\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768250 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-credential-socket\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768544 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-data\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768798 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-data\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.768922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.768848 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5339c3-a472-44fe-8300-83fdf9114bc5-istiod-ca-cert\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.770222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.770204 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-envoy\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.770490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.770471 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-podinfo\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.779942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.779920 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-token\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.780025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.779998 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdsb\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-kube-api-access-vhdsb\") pod \"e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:23.955789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:23.955702 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:24.246521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:24.246479 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" event={"ID":"1ae3bde8-b116-422a-92a7-2e23f76a71b2","Type":"ContainerStarted","Data":"705f8805edbcfb4000d683954bb8df36f2ce5b527cfbe24bc1df22901f8ae1c0"} Nov 04 01:20:24.552137 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:20:24.552106 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5339c3_a472_44fe_8300_83fdf9114bc5.slice/crio-93691ea89be392d1729c4d9ae33e172f3e09ae449c1b33e02f3fb88c7d5f2302 WatchSource:0}: Error finding container 93691ea89be392d1729c4d9ae33e172f3e09ae449c1b33e02f3fb88c7d5f2302: Status 404 returned error can't find the container with id 93691ea89be392d1729c4d9ae33e172f3e09ae449c1b33e02f3fb88c7d5f2302 Nov 04 01:20:24.553602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:24.553569 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx"] Nov 04 01:20:24.555248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:24.555216 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:24.555300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:24.555285 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:24.555367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:24.555311 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:25.196423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.196381 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" event={"ID":"6a5339c3-a472-44fe-8300-83fdf9114bc5","Type":"ContainerStarted","Data":"7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e"} Nov 04 01:20:25.196864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.196478 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" event={"ID":"6a5339c3-a472-44fe-8300-83fdf9114bc5","Type":"ContainerStarted","Data":"93691ea89be392d1729c4d9ae33e172f3e09ae449c1b33e02f3fb88c7d5f2302"} Nov 04 01:20:25.204310 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.204281 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" event={"ID":"1ae3bde8-b116-422a-92a7-2e23f76a71b2","Type":"ContainerStarted","Data":"503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae"} Nov 04 01:20:25.204470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.204377 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:25.225172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.225125 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podStartSLOduration=2.225111836 podStartE2EDuration="2.225111836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:25.220999603 +0000 UTC m=+1057.243931641" watchObservedRunningTime="2026-11-04 01:20:25.225111836 +0000 UTC m=+1057.248043842" Nov 04 01:20:25.241508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.241467 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" podStartSLOduration=2.796371797 podStartE2EDuration="3.241454975s" totalImagesPullingTime="445.083178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:25.240329772 +0000 UTC m=+1057.263261793" watchObservedRunningTime="2026-11-04 01:20:25.241454975 +0000 UTC m=+1057.264387026" Nov 04 01:20:25.956811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.956781 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:25.961809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:25.961784 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:26.212123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:26.212042 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:26.213041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:26.213018 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:20:27.972934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:27.971503 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:20:27.978360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:27.977506 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:20:29.669767 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:20:29.669728 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:20:31.217477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:31.217446 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:20:45.415482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:45.415425 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:45.420888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:45.420849 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:45.467501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:45.467452 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:45.944355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:45.944037 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 04 01:20:45.944580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:45.944421 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:20:46.224631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.224524 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:46.321663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.321602 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:46.475635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.475490 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:46.506588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.488441 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:46.524356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.522908 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:20:46.524356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.522974 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:20:46.963587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.963545 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:46.964353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:46.964310 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:20:47.008675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:47.008646 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:20:47.023997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:47.022074 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:20:47.026252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:47.025660 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:20:54.480220 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.480172 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk"] Nov 04 01:20:54.488162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.488135 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.500821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.500779 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e7742169-openshift-default-dockercfg-dp2rm\"" Nov 04 01:20:54.502244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.502214 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk"] Nov 04 01:20:54.504259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504230 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnwc\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-kube-api-access-vhnwc\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504382 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504425 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504462 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504534 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504617 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504650 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504725 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.504776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.504758 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0136259d-c35d-4e2d-98b9-8e1c76565d09-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.606832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.606797 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.606846 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0136259d-c35d-4e2d-98b9-8e1c76565d09-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.607036 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnwc\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-kube-api-access-vhnwc\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.607162 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.607541 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.607606 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.607661 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.607760 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.607793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.607761 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0136259d-c35d-4e2d-98b9-8e1c76565d09-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.608121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.608040 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.608121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.608051 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.608121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.608101 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.608491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.608469 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.608668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.608648 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.610099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.610080 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.610211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.610142 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.619492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.619464 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.619584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.619567 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnwc\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-kube-api-access-vhnwc\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-bqkxk\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:54.663981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.663948 4312 generic.go:356] "Generic (PLEG): container finished" podID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerID="10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a" exitCode=0 Nov 04 01:20:54.664178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.664029 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" event={"ID":"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa","Type":"ContainerDied","Data":"10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a"} Nov 04 01:20:54.664592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.664573 4312 scope.go:122] "RemoveContainer" containerID="10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a" Nov 04 01:20:54.666394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.666369 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:54.666453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.666433 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:54.666491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.666465 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:54.812558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:54.812526 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:55.252837 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:20:55.252797 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0136259d_c35d_4e2d_98b9_8e1c76565d09.slice/crio-84521e0756bcf4ab23e3f40329e50593ee734252f500f46f06a5bc9787324520 WatchSource:0}: Error finding container 84521e0756bcf4ab23e3f40329e50593ee734252f500f46f06a5bc9787324520: Status 404 returned error can't find the container with id 84521e0756bcf4ab23e3f40329e50593ee734252f500f46f06a5bc9787324520 Nov 04 01:20:55.253945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.253920 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk"] Nov 04 01:20:55.255954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.255922 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:55.256017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.255990 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:55.256060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.256019 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:20:55.721367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.720715 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" event={"ID":"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa","Type":"ContainerStarted","Data":"cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5"} Nov 04 01:20:55.727383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.724908 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" event={"ID":"0136259d-c35d-4e2d-98b9-8e1c76565d09","Type":"ContainerStarted","Data":"17fde16744070272d4ef9a459aa86054c9299ebf6e439a9882907f463c4b9f6d"} Nov 04 01:20:55.727383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.724995 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" event={"ID":"0136259d-c35d-4e2d-98b9-8e1c76565d09","Type":"ContainerStarted","Data":"84521e0756bcf4ab23e3f40329e50593ee734252f500f46f06a5bc9787324520"} Nov 04 01:20:55.795183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.794316 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podStartSLOduration=1.79429628 podStartE2EDuration="1.79429628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:55.792907088 +0000 UTC m=+1087.815839096" watchObservedRunningTime="2026-11-04 01:20:55.79429628 +0000 UTC m=+1087.817228286" Nov 04 01:20:55.812824 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.812789 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:20:55.945897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:55.945866 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:20:56.113011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:56.112977 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:20:56.114019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:56.113995 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:20:56.114131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:56.114052 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 01:20:56.818716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:56.818303 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 04 01:20:56.818716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:56.818374 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 04 01:20:57.184403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:57.159441 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:20:57.184403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:57.159503 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 01:20:57.813461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:57.813425 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 04 01:20:57.813621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:57.813480 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 04 01:20:58.146657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:58.136111 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:20:58.146657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:58.136180 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 01:20:58.813721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:58.813685 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 04 01:20:58.814010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:58.813982 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 04 01:20:59.112775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:59.112683 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.112775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:59.112744 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 01:20:59.505658 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:20:59.504572 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:20:59.826354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:59.818499 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.826354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:20:59.818561 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 04 01:21:00.113493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:00.113400 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.113493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:00.113481 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 01:21:00.819357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:00.818414 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.819357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:00.818476 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 04 01:21:01.112948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:01.112855 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.112948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:01.112917 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 04 01:21:01.812845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:01.812805 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.813051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:01.812865 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 04 01:21:02.117724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:02.117646 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:21:02.118848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:02.118827 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:21:02.818497 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:02.818467 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:21:02.818984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:02.818961 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:21:02.819654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:02.819635 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:21:12.061430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.061394 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk"] Nov 04 01:21:12.064965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.064943 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:12.082803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.082477 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0741ef\"" Nov 04 01:21:12.098780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.098746 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk"] Nov 04 01:21:12.099174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.099149 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0741ef-dockercfg-g4x6n\"" Nov 04 01:21:12.109403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.109360 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qrtm\" (UniqueName: \"kubernetes.io/projected/579850bb-cc33-4301-96b0-87cfeef3f110-kube-api-access-5qrtm\") pod \"payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:12.109678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.109660 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579850bb-cc33-4301-96b0-87cfeef3f110-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:12.141450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.141419 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv"] Nov 04 01:21:12.146078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.146055 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.147184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.147158 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv"] Nov 04 01:21:12.295643 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.295596 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlnb\" (UniqueName: \"kubernetes.io/projected/f20b9587-222a-4254-86f5-480e6c4a2feb-kube-api-access-6qlnb\") pod \"payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.310361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.303543 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20b9587-222a-4254-86f5-480e6c4a2feb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.329552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.323753 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qrtm\" (UniqueName: \"kubernetes.io/projected/579850bb-cc33-4301-96b0-87cfeef3f110-kube-api-access-5qrtm\") pod \"payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:12.387355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.380638 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579850bb-cc33-4301-96b0-87cfeef3f110-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:12.577354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.424288 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579850bb-cc33-4301-96b0-87cfeef3f110-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:12.577354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.521241 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlnb\" (UniqueName: \"kubernetes.io/projected/f20b9587-222a-4254-86f5-480e6c4a2feb-kube-api-access-6qlnb\") pod \"payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.577354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.521379 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20b9587-222a-4254-86f5-480e6c4a2feb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.577354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.522754 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20b9587-222a-4254-86f5-480e6c4a2feb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.597117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.597033 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlnb\" (UniqueName: \"kubernetes.io/projected/f20b9587-222a-4254-86f5-480e6c4a2feb-kube-api-access-6qlnb\") pod \"payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.620568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.620538 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qrtm\" (UniqueName: \"kubernetes.io/projected/579850bb-cc33-4301-96b0-87cfeef3f110-kube-api-access-5qrtm\") pod \"payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:12.790354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.784766 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:12.790354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:12.787539 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:13.110410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:13.109410 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv"] Nov 04 01:21:13.122390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:13.122306 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk"] Nov 04 01:21:13.148577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:13.148538 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" event={"ID":"f20b9587-222a-4254-86f5-480e6c4a2feb","Type":"ContainerStarted","Data":"e1902906d5aab915f05667ef36a04692682fb426ad70210ce5fa179761cecc1a"} Nov 04 01:21:13.150506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:13.150470 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" event={"ID":"579850bb-cc33-4301-96b0-87cfeef3f110","Type":"ContainerStarted","Data":"33ff8348bebc786378e67328bf3681912357f1dd47a41066933bd45e1a500590"} Nov 04 01:21:14.237360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:14.236814 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" event={"ID":"f20b9587-222a-4254-86f5-480e6c4a2feb","Type":"ContainerStarted","Data":"15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7"} Nov 04 01:21:14.241104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:14.237898 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:14.241104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:14.240514 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" event={"ID":"579850bb-cc33-4301-96b0-87cfeef3f110","Type":"ContainerStarted","Data":"4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a"} Nov 04 01:21:14.242829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:14.242033 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:14.261852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:14.261802 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" podStartSLOduration=2.261790037 podStartE2EDuration="2.261790037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:14.256629191 +0000 UTC m=+1106.279561219" watchObservedRunningTime="2026-11-04 01:21:14.261790037 +0000 UTC m=+1106.284722044" Nov 04 01:21:14.278265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:14.278211 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" podStartSLOduration=2.278197706 podStartE2EDuration="2.278197706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:14.274172699 +0000 UTC m=+1106.297104708" watchObservedRunningTime="2026-11-04 01:21:14.278197706 +0000 UTC m=+1106.301129713" Nov 04 01:21:16.257591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:16.257507 4312 generic.go:356] "Generic (PLEG): container finished" podID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerID="7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e" exitCode=0 Nov 04 01:21:16.257591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:16.257579 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" event={"ID":"6a5339c3-a472-44fe-8300-83fdf9114bc5","Type":"ContainerDied","Data":"7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e"} Nov 04 01:21:16.258111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:16.258067 4312 scope.go:122] "RemoveContainer" containerID="7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e" Nov 04 01:21:16.259815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:16.259787 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:21:16.259908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:16.259851 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:21:16.259908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:16.259885 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:21:17.295358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:17.292389 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" event={"ID":"6a5339c3-a472-44fe-8300-83fdf9114bc5","Type":"ContainerStarted","Data":"8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700"} Nov 04 01:21:17.968357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:17.964640 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:21:17.979362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:17.977475 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 01:21:17.979362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:17.977555 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 01:21:18.971356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:18.970800 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 01:21:18.971356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:18.970882 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 01:21:19.959365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:19.958919 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 01:21:19.959365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:19.958997 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 01:21:20.961236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:20.961191 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 01:21:20.974052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:20.973988 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 01:21:21.960895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:21.960853 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 01:21:21.961244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:21.961212 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 01:21:22.971370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:22.970733 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 04 01:21:22.971370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:22.970809 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 04 01:21:23.979357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:23.977157 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:21:23.979357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:23.979074 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:21:23.996163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:23.996126 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:21:25.251627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:25.251594 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:21:26.290812 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:26.290760 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:21:29.531799 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:21:29.531714 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:21:39.590836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.590800 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74"] Nov 04 01:21:39.601858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.601832 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.605371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.605324 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74"] Nov 04 01:21:39.743415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743325 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743514 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743557 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743657 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743691 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743768 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnpcz\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-kube-api-access-bnpcz\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743825 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743881 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61b2c5e-921a-441d-a727-a35a0fed31d1-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.743995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.743984 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845176 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845227 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845395 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpcz\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-kube-api-access-bnpcz\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845441 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845487 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61b2c5e-921a-441d-a727-a35a0fed31d1-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845577 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845662 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845682 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845812 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845880 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.845936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845917 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.846013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.845983 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.846287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.846263 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.846434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.846408 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61b2c5e-921a-441d-a727-a35a0fed31d1-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.847782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.847761 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.848321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.848305 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.857556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.857537 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.857931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.857909 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpcz\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-kube-api-access-bnpcz\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-8dw74\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:39.956989 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:39.956961 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:40.248731 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:21:40.248697 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b2c5e_921a_441d_a727_a35a0fed31d1.slice/crio-62a82dfe0855e539b7ee45c4a2922d1dc904c162eb13ae645e3dd5f58c6eec94 WatchSource:0}: Error finding container 62a82dfe0855e539b7ee45c4a2922d1dc904c162eb13ae645e3dd5f58c6eec94: Status 404 returned error can't find the container with id 62a82dfe0855e539b7ee45c4a2922d1dc904c162eb13ae645e3dd5f58c6eec94 Nov 04 01:21:40.249254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.249232 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74"] Nov 04 01:21:40.251603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.251569 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:21:40.251692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.251642 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:21:40.251692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.251689 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:21:40.798502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.798466 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" event={"ID":"b61b2c5e-921a-441d-a727-a35a0fed31d1","Type":"ContainerStarted","Data":"ad60383a04bf84ac66ac187fde586eab6409e4f4e49e877c28221f9539e17598"} Nov 04 01:21:40.798924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.798532 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" event={"ID":"b61b2c5e-921a-441d-a727-a35a0fed31d1","Type":"ContainerStarted","Data":"62a82dfe0855e539b7ee45c4a2922d1dc904c162eb13ae645e3dd5f58c6eec94"} Nov 04 01:21:40.828221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.828168 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podStartSLOduration=1.828155519 podStartE2EDuration="1.828155519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:40.822187503 +0000 UTC m=+1132.845119546" watchObservedRunningTime="2026-11-04 01:21:40.828155519 +0000 UTC m=+1132.851087525" Nov 04 01:21:40.966356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:40.958575 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:41.977238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:41.977202 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-8dw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:21:41.977813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:41.977779 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:21:42.964356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:42.960851 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-8dw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:21:42.964356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:42.960916 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:21:43.992384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:43.970725 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-8dw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:21:43.993008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:43.992969 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:21:44.962452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:44.962413 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-8dw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:21:44.962754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:44.962722 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:21:45.965360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:45.965308 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-8dw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.965982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:45.965952 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:21:46.646547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.646516 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb"] Nov 04 01:21:46.679856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.679826 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:46.682137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.682110 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb"] Nov 04 01:21:46.692541 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.692487 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-27bc4f8b-dockercfg-rg5qp\"" Nov 04 01:21:46.701501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.701476 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6"] Nov 04 01:21:46.707711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.707689 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:46.713072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.713052 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-27bc4f8b\"" Nov 04 01:21:46.714999 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.714933 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6"] Nov 04 01:21:46.726292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.726264 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5351dda-dd30-40fc-b844-c06fb755d763-plugins-config-volume\") pod \"payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:46.726466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.726447 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12f6c86-a8e5-4421-9b01-b85bae80d3da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:46.726511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.726497 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7c7\" (UniqueName: \"kubernetes.io/projected/e12f6c86-a8e5-4421-9b01-b85bae80d3da-kube-api-access-gw7c7\") pod \"payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:46.726611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.726586 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wln\" (UniqueName: \"kubernetes.io/projected/a5351dda-dd30-40fc-b844-c06fb755d763-kube-api-access-57wln\") pod \"payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:46.867364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.866446 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5351dda-dd30-40fc-b844-c06fb755d763-plugins-config-volume\") pod \"payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:46.867364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.866595 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12f6c86-a8e5-4421-9b01-b85bae80d3da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:46.867364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.866662 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7c7\" (UniqueName: \"kubernetes.io/projected/e12f6c86-a8e5-4421-9b01-b85bae80d3da-kube-api-access-gw7c7\") pod \"payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:46.867364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.866705 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wln\" (UniqueName: \"kubernetes.io/projected/a5351dda-dd30-40fc-b844-c06fb755d763-kube-api-access-57wln\") pod \"payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:46.881434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.868012 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5351dda-dd30-40fc-b844-c06fb755d763-plugins-config-volume\") pod \"payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:46.986411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.985471 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-8dw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.013357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.987126 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:21:47.013357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:46.989302 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12f6c86-a8e5-4421-9b01-b85bae80d3da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:47.023285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.019181 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7c7\" (UniqueName: \"kubernetes.io/projected/e12f6c86-a8e5-4421-9b01-b85bae80d3da-kube-api-access-gw7c7\") pod \"payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:47.031362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.030558 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wln\" (UniqueName: \"kubernetes.io/projected/a5351dda-dd30-40fc-b844-c06fb755d763-kube-api-access-57wln\") pod \"payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:47.032364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.032192 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:47.308125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.308088 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:47.641564 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:21:47.633955 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5351dda_dd30_40fc_b844_c06fb755d763.slice/crio-92fd3ff879177a592f1e44eda0cfbe55eb67384f8905dfdfce2cdc516ee83663 WatchSource:0}: Error finding container 92fd3ff879177a592f1e44eda0cfbe55eb67384f8905dfdfce2cdc516ee83663: Status 404 returned error can't find the container with id 92fd3ff879177a592f1e44eda0cfbe55eb67384f8905dfdfce2cdc516ee83663 Nov 04 01:21:47.678514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.678480 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6"] Nov 04 01:21:47.853782 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:21:47.853752 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12f6c86_a8e5_4421_9b01_b85bae80d3da.slice/crio-bd25ca7b00be48823372ff6b346fd6bd3226f5ad7db3d1ae7d91ebed75a18629 WatchSource:0}: Error finding container bd25ca7b00be48823372ff6b346fd6bd3226f5ad7db3d1ae7d91ebed75a18629: Status 404 returned error can't find the container with id bd25ca7b00be48823372ff6b346fd6bd3226f5ad7db3d1ae7d91ebed75a18629 Nov 04 01:21:47.861672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.857805 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb"] Nov 04 01:21:47.957637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.957612 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-8dw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.957754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:47.957654 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:21:48.187759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.187668 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" event={"ID":"e12f6c86-a8e5-4421-9b01-b85bae80d3da","Type":"ContainerStarted","Data":"bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d"} Nov 04 01:21:48.187759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.187727 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" event={"ID":"e12f6c86-a8e5-4421-9b01-b85bae80d3da","Type":"ContainerStarted","Data":"bd25ca7b00be48823372ff6b346fd6bd3226f5ad7db3d1ae7d91ebed75a18629"} Nov 04 01:21:48.187759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.187754 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:48.189519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.189488 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" event={"ID":"a5351dda-dd30-40fc-b844-c06fb755d763","Type":"ContainerStarted","Data":"847b8540eddf14b7f62df4f895426b8edbc65f7f0050307cb670ee2420121e20"} Nov 04 01:21:48.189657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.189539 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" event={"ID":"a5351dda-dd30-40fc-b844-c06fb755d763","Type":"ContainerStarted","Data":"92fd3ff879177a592f1e44eda0cfbe55eb67384f8905dfdfce2cdc516ee83663"} Nov 04 01:21:48.189657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.189582 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:48.210719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.210667 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" podStartSLOduration=2.210649588 podStartE2EDuration="2.210649588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:48.205288183 +0000 UTC m=+1140.228220190" watchObservedRunningTime="2026-11-04 01:21:48.210649588 +0000 UTC m=+1140.233581595" Nov 04 01:21:48.229122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.229075 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" podStartSLOduration=2.229063107 podStartE2EDuration="2.229063107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:48.225952207 +0000 UTC m=+1140.248884238" watchObservedRunningTime="2026-11-04 01:21:48.229063107 +0000 UTC m=+1140.251995114" Nov 04 01:21:48.961799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.961766 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:48.962274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.962250 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:48.963032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:48.963014 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:21:49.183357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:49.182725 4312 scope.go:122] "RemoveContainer" containerID="83081907dda0291bf099a8ae7be45449afae08fc8fa1458a2291ca52236677db" Nov 04 01:21:49.237630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:49.232227 4312 scope.go:122] "RemoveContainer" containerID="40e4979a8f0cb6d5e4001f3fee39c97540db4031dd6b6f3c9fb4f0c9edf1185e" Nov 04 01:21:49.266545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:49.263876 4312 scope.go:122] "RemoveContainer" containerID="00376732d643b0bb7712f672e0ba3268732aac61d75e9160766a6641b8e822d5" Nov 04 01:21:51.940959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:51.940926 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5"] Nov 04 01:21:51.948453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:51.948426 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:51.966567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:51.963354 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5"] Nov 04 01:21:51.972414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:51.972388 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-5ce9e8-serving-cert\"" Nov 04 01:21:52.107174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.094997 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.111502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.111469 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.111502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.111565 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbtdb\" (UniqueName: \"kubernetes.io/projected/1729a5d9-9158-4167-a20f-033ed229a473-kube-api-access-jbtdb\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.111925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.111730 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1729a5d9-9158-4167-a20f-033ed229a473-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.226407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.218013 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.226407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.218321 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.226407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.218591 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbtdb\" (UniqueName: \"kubernetes.io/projected/1729a5d9-9158-4167-a20f-033ed229a473-kube-api-access-jbtdb\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.226407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.218733 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1729a5d9-9158-4167-a20f-033ed229a473-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.226407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.220012 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1729a5d9-9158-4167-a20f-033ed229a473-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.250198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.244317 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.250198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.248593 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.257230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.257201 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbtdb\" (UniqueName: \"kubernetes.io/projected/1729a5d9-9158-4167-a20f-033ed229a473-kube-api-access-jbtdb\") pod \"maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.273276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.273250 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:52.703456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:52.703326 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5"] Nov 04 01:21:53.345368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:53.345303 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" event={"ID":"1729a5d9-9158-4167-a20f-033ed229a473","Type":"ContainerStarted","Data":"10cc118626521a3e41f3b240ab82d0e799ad2a7f8ea34e85d07f2c74806f8466"} Nov 04 01:21:54.354825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:54.354783 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" event={"ID":"1729a5d9-9158-4167-a20f-033ed229a473","Type":"ContainerStarted","Data":"2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15"} Nov 04 01:21:54.355262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:54.354855 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:21:54.379071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:54.379020 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" podStartSLOduration=2.727008841 podStartE2EDuration="3.3790055s" totalImagesPullingTime="651.996659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:54.374903288 +0000 UTC m=+1146.397835296" watchObservedRunningTime="2026-11-04 01:21:54.3790055 +0000 UTC m=+1146.401937507" Nov 04 01:21:59.207482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:59.207454 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:21:59.207942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:21:59.207535 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:21:59.582309 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:21:59.582262 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:22:00.382794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:22:00.382761 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:22:29.568859 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:22:29.568820 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:22:51.026529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:22:51.026497 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr"] Nov 04 01:22:51.034687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:22:51.034656 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534009bf-xkfkr"] Nov 04 01:22:52.110712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:22:52.110681 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a697aba5-b71b-499a-b06f-43f635f8b862" path="/var/lib/kubelet/pods/a697aba5-b71b-499a-b06f-43f635f8b862/volumes" Nov 04 01:22:59.549086 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:22:59.549002 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:23:02.131319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.131286 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx"] Nov 04 01:23:02.137823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.137796 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.145790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.145216 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6e609c-serving-cert\"" Nov 04 01:23:02.146676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.146651 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx"] Nov 04 01:23:02.251515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.251470 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-openshift-service-ca\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.251986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.251960 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-metrics-certs\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.252210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.252193 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bm46\" (UniqueName: \"kubernetes.io/projected/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-kube-api-access-2bm46\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.252463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.252445 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-maas-api-tls\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.356635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.356593 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-openshift-service-ca\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.358097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.358069 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-metrics-certs\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.362953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.362925 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bm46\" (UniqueName: \"kubernetes.io/projected/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-kube-api-access-2bm46\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.363555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.363530 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-maas-api-tls\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.374362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.357885 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-openshift-service-ca\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.453480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.433421 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-maas-api-tls\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.453480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.434026 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-metrics-certs\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.554036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.553996 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bm46\" (UniqueName: \"kubernetes.io/projected/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-kube-api-access-2bm46\") pod \"maas-api-e2e-api-a-6e609c-d7477dd4-88fqx\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:02.781356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:02.780664 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:03.331893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:03.331855 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx"] Nov 04 01:23:03.628241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:03.628196 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" event={"ID":"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5","Type":"ContainerStarted","Data":"befac66a18186cd459077a730b981e7cb50c2e8b2e515a5e1300f63f11e9c3fe"} Nov 04 01:23:04.637405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:04.637368 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" event={"ID":"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5","Type":"ContainerStarted","Data":"664b7d9fa9107a68ad4312302e725caf5556217cdf587c6861df8ef08d84bbd1"} Nov 04 01:23:04.637890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:04.637435 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:04.657739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:04.657677 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" podStartSLOduration=1.9834043540000001 podStartE2EDuration="2.657656761s" totalImagesPullingTime="674.252407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:04.656814926 +0000 UTC m=+1216.679746933" watchObservedRunningTime="2026-11-04 01:23:04.657656761 +0000 UTC m=+1216.680588769" Nov 04 01:23:10.651245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:10.651217 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:23:29.443083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.443046 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp"] Nov 04 01:23:29.457483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.456396 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:29.460669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.460639 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp"] Nov 04 01:23:29.466321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.466294 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-20a250-dockercfg-xfhbk\"" Nov 04 01:23:29.481919 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:29.481884 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:23:29.515696 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.514058 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-20a250\"" Nov 04 01:23:29.588355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.581947 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3901502-f9b2-4a22-aba1-43a7947e7047-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:29.588355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.582180 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwxt\" (UniqueName: \"kubernetes.io/projected/e3901502-f9b2-4a22-aba1-43a7947e7047-kube-api-access-qvwxt\") pod \"payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:29.761416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.750960 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3901502-f9b2-4a22-aba1-43a7947e7047-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:29.761416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.751179 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwxt\" (UniqueName: \"kubernetes.io/projected/e3901502-f9b2-4a22-aba1-43a7947e7047-kube-api-access-qvwxt\") pod \"payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:29.761416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.752730 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3901502-f9b2-4a22-aba1-43a7947e7047-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:29.879407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:29.878125 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwxt\" (UniqueName: \"kubernetes.io/projected/e3901502-f9b2-4a22-aba1-43a7947e7047-kube-api-access-qvwxt\") pod \"payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:30.083836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.083805 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:30.384453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.384407 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp"] Nov 04 01:23:30.559037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.559009 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv"] Nov 04 01:23:30.564256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.564231 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:30.570016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.569906 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fdd516-dockercfg-mrkr7\"" Nov 04 01:23:30.570603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.570564 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fdd516\"" Nov 04 01:23:30.576131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.575442 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv"] Nov 04 01:23:30.576131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.576105 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e617c76-3e92-4f95-81b0-e0a9977f38a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:30.576244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.576146 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t642l\" (UniqueName: \"kubernetes.io/projected/2e617c76-3e92-4f95-81b0-e0a9977f38a1-kube-api-access-t642l\") pod \"payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:30.601380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.601348 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km"] Nov 04 01:23:30.612819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.612787 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:30.620732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:30.619393 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km"] Nov 04 01:23:31.359360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.341815 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e617c76-3e92-4f95-81b0-e0a9977f38a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:31.359360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.341865 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t642l\" (UniqueName: \"kubernetes.io/projected/2e617c76-3e92-4f95-81b0-e0a9977f38a1-kube-api-access-t642l\") pod \"payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:31.359360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.341931 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l795s\" (UniqueName: \"kubernetes.io/projected/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-kube-api-access-l795s\") pod \"payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:31.359360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.342222 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:31.359360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.351303 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e617c76-3e92-4f95-81b0-e0a9977f38a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:31.427792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.422212 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t642l\" (UniqueName: \"kubernetes.io/projected/2e617c76-3e92-4f95-81b0-e0a9977f38a1-kube-api-access-t642l\") pod \"payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:31.476372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.448541 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l795s\" (UniqueName: \"kubernetes.io/projected/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-kube-api-access-l795s\") pod \"payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:31.476372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.448880 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:31.476372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.453620 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:31.488892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.488842 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" event={"ID":"e3901502-f9b2-4a22-aba1-43a7947e7047","Type":"ContainerStarted","Data":"fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904"} Nov 04 01:23:31.489195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.489172 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" event={"ID":"e3901502-f9b2-4a22-aba1-43a7947e7047","Type":"ContainerStarted","Data":"4cc63784325e519d091294d404404be1cade447ba25157e94cab2f03e8a75dd0"} Nov 04 01:23:31.491789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.491542 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:31.514480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.494123 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:31.672425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.668860 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l795s\" (UniqueName: \"kubernetes.io/projected/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-kube-api-access-l795s\") pod \"payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:31.703042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.702965 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" podStartSLOduration=2.702942653 podStartE2EDuration="2.702942653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:31.523114484 +0000 UTC m=+1243.546046502" watchObservedRunningTime="2026-11-04 01:23:31.702942653 +0000 UTC m=+1243.725874660" Nov 04 01:23:31.916619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:31.916389 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:32.164894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.164415 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94"] Nov 04 01:23:32.164894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.164478 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm"] Nov 04 01:23:32.360209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.360166 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv"] Nov 04 01:23:32.661365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.657504 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" event={"ID":"2e617c76-3e92-4f95-81b0-e0a9977f38a1","Type":"ContainerStarted","Data":"fe5f9e8497f2443c751987a0f74c7a5701929c8e0117884f5bedccae6461e761"} Nov 04 01:23:32.661365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.659405 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km"] Nov 04 01:23:32.962995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.962049 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z"] Nov 04 01:23:32.962995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.962316 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" containerID="cri-o://eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95" gracePeriod=30 Nov 04 01:23:32.971580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.970161 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg"] Nov 04 01:23:32.971580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:32.970421 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" containerID="cri-o://7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a" gracePeriod=30 Nov 04 01:23:33.531348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.531189 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd"] Nov 04 01:23:33.531524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.531479 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" containerID="cri-o://9dbeea53531ac47451b81cb904738b6dd5f15aed6bd43efc062373fc51c907a9" gracePeriod=30 Nov 04 01:23:33.539604 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.539573 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz"] Nov 04 01:23:33.539826 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.539795 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" containerID="cri-o://b2bd8dc05bd8173e5e91e6efbd502af2fcf631be056a9973112bd19f232e6ffa" gracePeriod=30 Nov 04 01:23:33.665828 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.665791 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" event={"ID":"2e617c76-3e92-4f95-81b0-e0a9977f38a1","Type":"ContainerStarted","Data":"0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930"} Nov 04 01:23:33.666010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.665859 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:33.667551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.667523 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" event={"ID":"2bc27530-aef3-4e8d-83b7-ebc205aa50ec","Type":"ContainerStarted","Data":"b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9"} Nov 04 01:23:33.667710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.667568 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" event={"ID":"2bc27530-aef3-4e8d-83b7-ebc205aa50ec","Type":"ContainerStarted","Data":"53a687d81fea325e1ebea5a773e7de5f86f433f014d43c2bc6a94f10de1c8367"} Nov 04 01:23:33.667768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.667729 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:33.686201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.686149 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" podStartSLOduration=3.686136032 podStartE2EDuration="3.686136032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:33.683448562 +0000 UTC m=+1245.706380611" watchObservedRunningTime="2026-11-04 01:23:33.686136032 +0000 UTC m=+1245.709068039" Nov 04 01:23:33.706374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:33.706316 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" podStartSLOduration=3.706302591 podStartE2EDuration="3.706302591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:33.700855818 +0000 UTC m=+1245.723787844" watchObservedRunningTime="2026-11-04 01:23:33.706302591 +0000 UTC m=+1245.729234641" Nov 04 01:23:37.164786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.164740 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" podUID="faa96b9c-c094-4d25-b5e0-de31751312e3" containerName="payload-pre-processing" containerID="cri-o://faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456" gracePeriod=25 Nov 04 01:23:37.166232 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.166193 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" podUID="1b880e25-057e-4833-9dcb-a2c529abeb64" containerName="payload-processing" containerID="cri-o://a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba" gracePeriod=25 Nov 04 01:23:37.612896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.612858 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:23:37.644343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.644301 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:23:37.695735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.695705 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa96b9c-c094-4d25-b5e0-de31751312e3-kube-api-access-m4x82\" (UniqueName: \"kubernetes.io/projected/faa96b9c-c094-4d25-b5e0-de31751312e3-kube-api-access-m4x82\") pod \"faa96b9c-c094-4d25-b5e0-de31751312e3\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " Nov 04 01:23:37.695938 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.695921 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa96b9c-c094-4d25-b5e0-de31751312e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa96b9c-c094-4d25-b5e0-de31751312e3-plugins-config-volume\") pod \"faa96b9c-c094-4d25-b5e0-de31751312e3\" (UID: \"faa96b9c-c094-4d25-b5e0-de31751312e3\") " Nov 04 01:23:37.696434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.696399 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa96b9c-c094-4d25-b5e0-de31751312e3-plugins-config-volume" pod "faa96b9c-c094-4d25-b5e0-de31751312e3" (UID: "faa96b9c-c094-4d25-b5e0-de31751312e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:37.697046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.697004 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa96b9c-c094-4d25-b5e0-de31751312e3-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:37.698586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.698562 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa96b9c-c094-4d25-b5e0-de31751312e3-kube-api-access-m4x82" pod "faa96b9c-c094-4d25-b5e0-de31751312e3" (UID: "faa96b9c-c094-4d25-b5e0-de31751312e3"). InnerVolumeSpecName "kube-api-access-m4x82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:37.701638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.701608 4312 generic.go:356] "Generic (PLEG): container finished" podID="1b880e25-057e-4833-9dcb-a2c529abeb64" containerID="a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba" exitCode=0 Nov 04 01:23:37.701752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.701654 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" Nov 04 01:23:37.701752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.701694 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" event={"ID":"1b880e25-057e-4833-9dcb-a2c529abeb64","Type":"ContainerDied","Data":"a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba"} Nov 04 01:23:37.701752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.701750 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94" event={"ID":"1b880e25-057e-4833-9dcb-a2c529abeb64","Type":"ContainerDied","Data":"66dfa6c5df118a56f7fd1f4be0645b32ba853448f0d8db26bd2f318e7794efc0"} Nov 04 01:23:37.701899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.701772 4312 scope.go:122] "RemoveContainer" containerID="a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba" Nov 04 01:23:37.703797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.703773 4312 generic.go:356] "Generic (PLEG): container finished" podID="faa96b9c-c094-4d25-b5e0-de31751312e3" containerID="faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456" exitCode=0 Nov 04 01:23:37.703889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.703815 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" event={"ID":"faa96b9c-c094-4d25-b5e0-de31751312e3","Type":"ContainerDied","Data":"faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456"} Nov 04 01:23:37.703889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.703825 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" Nov 04 01:23:37.703889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.703856 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm" event={"ID":"faa96b9c-c094-4d25-b5e0-de31751312e3","Type":"ContainerDied","Data":"1aea6450e13943c7d78c081b838d78e82d24e6f0e56a09023c2b41a1e5f7d4a4"} Nov 04 01:23:37.724621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.723444 4312 scope.go:122] "RemoveContainer" containerID="a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba" Nov 04 01:23:37.724621 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:37.723993 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba\": container with ID starting with a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba not found: ID does not exist" containerID="a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba" Nov 04 01:23:37.724621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.724030 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba"} err="failed to get container status \"a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba\": rpc error: code = NotFound desc = could not find container \"a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba\": container with ID starting with a4bf0b3a3c9c69642e380eb68ad6244b0c05d2f0c3de6b7ffd230546aeb7d1ba not found: ID does not exist" Nov 04 01:23:37.724621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.724052 4312 scope.go:122] "RemoveContainer" containerID="faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456" Nov 04 01:23:37.733423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.733397 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm"] Nov 04 01:23:37.736771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.736750 4312 scope.go:122] "RemoveContainer" containerID="faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456" Nov 04 01:23:37.737053 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:37.737023 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456\": container with ID starting with faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456 not found: ID does not exist" containerID="faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456" Nov 04 01:23:37.737131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.737055 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456"} err="failed to get container status \"faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456\": rpc error: code = NotFound desc = could not find container \"faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456\": container with ID starting with faa2ba7472d10fdfbb0c0ffb66eeed3407da411e416dab02d861545944ccd456 not found: ID does not exist" Nov 04 01:23:37.738247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.738223 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7742169-6484bfc755-gfkcm"] Nov 04 01:23:37.798089 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.798062 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b880e25-057e-4833-9dcb-a2c529abeb64-kube-api-access-dffvn\" (UniqueName: \"kubernetes.io/projected/1b880e25-057e-4833-9dcb-a2c529abeb64-kube-api-access-dffvn\") pod \"1b880e25-057e-4833-9dcb-a2c529abeb64\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " Nov 04 01:23:37.798304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.798291 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b880e25-057e-4833-9dcb-a2c529abeb64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b880e25-057e-4833-9dcb-a2c529abeb64-plugins-config-volume\") pod \"1b880e25-057e-4833-9dcb-a2c529abeb64\" (UID: \"1b880e25-057e-4833-9dcb-a2c529abeb64\") " Nov 04 01:23:37.798667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.798644 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b880e25-057e-4833-9dcb-a2c529abeb64-plugins-config-volume" pod "1b880e25-057e-4833-9dcb-a2c529abeb64" (UID: "1b880e25-057e-4833-9dcb-a2c529abeb64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:37.799025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.799009 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b880e25-057e-4833-9dcb-a2c529abeb64-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:37.799067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.799030 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4x82\" (UniqueName: \"kubernetes.io/projected/faa96b9c-c094-4d25-b5e0-de31751312e3-kube-api-access-m4x82\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:37.800210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.800190 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b880e25-057e-4833-9dcb-a2c529abeb64-kube-api-access-dffvn" pod "1b880e25-057e-4833-9dcb-a2c529abeb64" (UID: "1b880e25-057e-4833-9dcb-a2c529abeb64"). InnerVolumeSpecName "kube-api-access-dffvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:37.901036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:37.901004 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dffvn\" (UniqueName: \"kubernetes.io/projected/1b880e25-057e-4833-9dcb-a2c529abeb64-kube-api-access-dffvn\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.135641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.135600 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa96b9c-c094-4d25-b5e0-de31751312e3" path="/var/lib/kubelet/pods/faa96b9c-c094-4d25-b5e0-de31751312e3/volumes" Nov 04 01:23:38.136237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.136213 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94"] Nov 04 01:23:38.136428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.136254 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7742169-f6b77d9d-5lm94"] Nov 04 01:23:38.433552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.433520 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:23:38.454643 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.454614 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-credential-socket\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.454797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.454697 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-kube-api-access-hwtc7\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-kube-api-access-hwtc7\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.454797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.454786 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-socket\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.454896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.454842 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-data\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.454896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.454876 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-podinfo\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.454977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.454907 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-credential-socket" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.454977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.454937 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551f2bc6-f790-4c9b-af7a-c183240e3b68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f2bc6-f790-4c9b-af7a-c183240e3b68-istiod-ca-cert\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.455097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.455077 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-socket" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.455159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.455093 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-certs\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.455159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.455125 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-data" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.455271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.455165 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-token\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-token\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.455271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.455215 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-envoy\") pod \"551f2bc6-f790-4c9b-af7a-c183240e3b68\" (UID: \"551f2bc6-f790-4c9b-af7a-c183240e3b68\") " Nov 04 01:23:38.455465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.455437 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-certs" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.455529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.455463 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551f2bc6-f790-4c9b-af7a-c183240e3b68-istiod-ca-cert" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:38.456206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.456177 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.456343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.456210 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.456343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.456226 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.456343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.456241 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f2bc6-f790-4c9b-af7a-c183240e3b68-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.456343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.456256 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.457238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.457214 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-kube-api-access-hwtc7" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "kube-api-access-hwtc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.457627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.457601 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-token" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.457795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.457686 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-podinfo" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:23:38.458152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.458088 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-envoy" pod "551f2bc6-f790-4c9b-af7a-c183240e3b68" (UID: "551f2bc6-f790-4c9b-af7a-c183240e3b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.521853 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.521821 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:23:38.557459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557422 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-certs\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.557640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557528 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-envoy\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.557640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557606 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-data\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.557778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557679 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-kube-api-access-j8wzr\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-kube-api-access-j8wzr\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.557778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557687 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-certs" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.557893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557861 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-token\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-token\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.557962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557936 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-credential-socket\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.558096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.557981 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-socket\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.558096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.558048 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-podinfo\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.558261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.558126 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836f5631-21d0-4b71-99cd-1d0a82c8a855-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836f5631-21d0-4b71-99cd-1d0a82c8a855-istiod-ca-cert\") pod \"836f5631-21d0-4b71-99cd-1d0a82c8a855\" (UID: \"836f5631-21d0-4b71-99cd-1d0a82c8a855\") " Nov 04 01:23:38.558622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.558597 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-data" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.558823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.558801 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-credential-socket" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.558916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.558816 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-socket" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.559465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559197 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836f5631-21d0-4b71-99cd-1d0a82c8a855-istiod-ca-cert" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559855 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559887 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwtc7\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-kube-api-access-hwtc7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559904 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559917 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559931 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559945 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836f5631-21d0-4b71-99cd-1d0a82c8a855-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559958 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559970 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f2bc6-f790-4c9b-af7a-c183240e3b68-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.560005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.559983 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.561026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.561000 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-token" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.561118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.561045 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-kube-api-access-j8wzr" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "kube-api-access-j8wzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.567225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.567192 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-envoy" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.568756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.568731 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-podinfo" pod "836f5631-21d0-4b71-99cd-1d0a82c8a855" (UID: "836f5631-21d0-4b71-99cd-1d0a82c8a855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:23:38.668237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.668197 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8wzr\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-kube-api-access-j8wzr\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.668237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.668239 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.668237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.668254 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.668237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.668267 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836f5631-21d0-4b71-99cd-1d0a82c8a855-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.718209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.718173 4312 generic.go:356] "Generic (PLEG): container finished" podID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerID="9dbeea53531ac47451b81cb904738b6dd5f15aed6bd43efc062373fc51c907a9" exitCode=0 Nov 04 01:23:38.718416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.718245 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" event={"ID":"af31fac3-3297-4d81-9439-0fef57db1aa0","Type":"ContainerDied","Data":"9dbeea53531ac47451b81cb904738b6dd5f15aed6bd43efc062373fc51c907a9"} Nov 04 01:23:38.721031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.721007 4312 generic.go:356] "Generic (PLEG): container finished" podID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerID="b2bd8dc05bd8173e5e91e6efbd502af2fcf631be056a9973112bd19f232e6ffa" exitCode=0 Nov 04 01:23:38.721181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.721071 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" event={"ID":"a1355eaa-1dac-49d2-9dab-31ba22187e0e","Type":"ContainerDied","Data":"b2bd8dc05bd8173e5e91e6efbd502af2fcf631be056a9973112bd19f232e6ffa"} Nov 04 01:23:38.723413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.723388 4312 generic.go:356] "Generic (PLEG): container finished" podID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerID="eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95" exitCode=0 Nov 04 01:23:38.723653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.723438 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" Nov 04 01:23:38.723800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.723454 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" event={"ID":"551f2bc6-f790-4c9b-af7a-c183240e3b68","Type":"ContainerDied","Data":"eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95"} Nov 04 01:23:38.723862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.723837 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z" event={"ID":"551f2bc6-f790-4c9b-af7a-c183240e3b68","Type":"ContainerDied","Data":"f2be21a782fbca09586beb8bc0d864271f3fc32546b69ea1cc712973cd0e150f"} Nov 04 01:23:38.723901 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.723875 4312 scope.go:122] "RemoveContainer" containerID="eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95" Nov 04 01:23:38.726499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.726477 4312 generic.go:356] "Generic (PLEG): container finished" podID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerID="7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a" exitCode=0 Nov 04 01:23:38.726585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.726525 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" event={"ID":"836f5631-21d0-4b71-99cd-1d0a82c8a855","Type":"ContainerDied","Data":"7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a"} Nov 04 01:23:38.726585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.726552 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" event={"ID":"836f5631-21d0-4b71-99cd-1d0a82c8a855","Type":"ContainerDied","Data":"6e45367f049073e0944918a995d64399b48fc0a86731e013668c9879ef13fa72"} Nov 04 01:23:38.726658 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.726615 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg" Nov 04 01:23:38.760058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.757187 4312 scope.go:122] "RemoveContainer" containerID="eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95" Nov 04 01:23:38.760058 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:38.758111 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95\": container with ID starting with eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95 not found: ID does not exist" containerID="eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95" Nov 04 01:23:38.760058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.758146 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95"} err="failed to get container status \"eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95\": rpc error: code = NotFound desc = could not find container \"eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95\": container with ID starting with eb1bc0845513eb53a7d12ccb1e6657171d966f8b11d0f2c195d3a73a2420cb95 not found: ID does not exist" Nov 04 01:23:38.760058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.758173 4312 scope.go:122] "RemoveContainer" containerID="7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a" Nov 04 01:23:38.762084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.762062 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z"] Nov 04 01:23:38.771101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.771074 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-wn82z"] Nov 04 01:23:38.783692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.783668 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg"] Nov 04 01:23:38.784499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.784199 4312 scope.go:122] "RemoveContainer" containerID="7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a" Nov 04 01:23:38.784776 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:38.784692 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a\": container with ID starting with 7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a not found: ID does not exist" containerID="7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a" Nov 04 01:23:38.784776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.784726 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a"} err="failed to get container status \"7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a\": rpc error: code = NotFound desc = could not find container \"7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a\": container with ID starting with 7996c59cafbbeaec94035ebc01387e402468ea305241fdd9a34bfdd3ff62a43a not found: ID does not exist" Nov 04 01:23:38.792584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.792564 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-s75zg"] Nov 04 01:23:38.835307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.835279 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:23:38.873233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.873200 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-certs\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.873576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.873556 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-data\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.873713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.873699 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-podinfo\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.873873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.873860 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-credential-socket\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.874112 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.874098 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af31fac3-3297-4d81-9439-0fef57db1aa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31fac3-3297-4d81-9439-0fef57db1aa0-istiod-ca-cert\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.874221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.874207 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-socket\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.874394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.874378 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-kube-api-access-5qw24\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-kube-api-access-5qw24\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.874637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.874623 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-envoy\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.874879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.874850 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-data" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.874962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.874942 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-credential-socket" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.875027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.874992 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-token\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-token\") pod \"af31fac3-3297-4d81-9439-0fef57db1aa0\" (UID: \"af31fac3-3297-4d81-9439-0fef57db1aa0\") " Nov 04 01:23:38.876945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.875680 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-certs" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.876945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.876052 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.876945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.876074 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.876945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.876088 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.876945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.876911 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af31fac3-3297-4d81-9439-0fef57db1aa0-istiod-ca-cert" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:38.877759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.877738 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-socket" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.879619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.879587 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-token" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.880957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.880826 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-kube-api-access-5qw24" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "kube-api-access-5qw24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.883390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.883351 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-podinfo" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:23:38.885014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.884958 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-envoy" pod "af31fac3-3297-4d81-9439-0fef57db1aa0" (UID: "af31fac3-3297-4d81-9439-0fef57db1aa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.911077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.911045 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:23:38.976885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.976831 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-podinfo\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.976904 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-kube-api-access-lph6b\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-kube-api-access-lph6b\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977002 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-envoy\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977059 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-credential-socket\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977103 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-token\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977205 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istiod-ca-cert\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977298 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-certs\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977356 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-credential-socket" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.977545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977470 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-data\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977604 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-socket\") pod \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\" (UID: \"a1355eaa-1dac-49d2-9dab-31ba22187e0e\") " Nov 04 01:23:38.977735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977608 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istiod-ca-cert" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:38.977795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977631 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-certs" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.977850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977790 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-data" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.977906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.977870 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-socket" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:38.978606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978576 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qw24\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-kube-api-access-5qw24\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978610 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978628 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978643 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978658 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978671 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978686 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978700 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31fac3-3297-4d81-9439-0fef57db1aa0-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978712 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978726 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31fac3-3297-4d81-9439-0fef57db1aa0-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.978929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.978740 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31fac3-3297-4d81-9439-0fef57db1aa0-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:38.979831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.979803 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-podinfo" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:23:38.979831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.979823 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-token" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.980141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.980108 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-kube-api-access-lph6b" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "kube-api-access-lph6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:38.980278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:38.980254 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-envoy" pod "a1355eaa-1dac-49d2-9dab-31ba22187e0e" (UID: "a1355eaa-1dac-49d2-9dab-31ba22187e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:23:39.088461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.088321 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:39.088461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.088374 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:39.088461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.088390 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lph6b\" (UniqueName: \"kubernetes.io/projected/a1355eaa-1dac-49d2-9dab-31ba22187e0e-kube-api-access-lph6b\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:39.088461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.088405 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1355eaa-1dac-49d2-9dab-31ba22187e0e-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:39.791361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.784327 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" Nov 04 01:23:39.791361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.787606 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz" event={"ID":"a1355eaa-1dac-49d2-9dab-31ba22187e0e","Type":"ContainerDied","Data":"63a4a4a9f82e04de0edbbda69dd871e2305ff5b91356f4bc56767d05e98b4532"} Nov 04 01:23:39.791361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.787768 4312 scope.go:122] "RemoveContainer" containerID="b2bd8dc05bd8173e5e91e6efbd502af2fcf631be056a9973112bd19f232e6ffa" Nov 04 01:23:39.802378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.798949 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" event={"ID":"af31fac3-3297-4d81-9439-0fef57db1aa0","Type":"ContainerDied","Data":"46d9c98e6b538a00d5f07fbec516e8ec998a51dcd55fe578f96c10968ce5db15"} Nov 04 01:23:39.802378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.799273 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd" Nov 04 01:23:39.848801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.848783 4312 scope.go:122] "RemoveContainer" containerID="9dbeea53531ac47451b81cb904738b6dd5f15aed6bd43efc062373fc51c907a9" Nov 04 01:23:39.858154 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.858124 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz"] Nov 04 01:23:39.863663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.863633 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-n7jzz"] Nov 04 01:23:39.878167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.878145 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd"] Nov 04 01:23:39.883181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:39.883160 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-95wwd"] Nov 04 01:23:40.169970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:40.169889 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b880e25-057e-4833-9dcb-a2c529abeb64" path="/var/lib/kubelet/pods/1b880e25-057e-4833-9dcb-a2c529abeb64/volumes" Nov 04 01:23:40.171044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:40.171019 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" path="/var/lib/kubelet/pods/551f2bc6-f790-4c9b-af7a-c183240e3b68/volumes" Nov 04 01:23:40.172271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:40.172245 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" path="/var/lib/kubelet/pods/836f5631-21d0-4b71-99cd-1d0a82c8a855/volumes" Nov 04 01:23:40.179949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:40.179924 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" path="/var/lib/kubelet/pods/a1355eaa-1dac-49d2-9dab-31ba22187e0e/volumes" Nov 04 01:23:40.181010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:40.180987 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" path="/var/lib/kubelet/pods/af31fac3-3297-4d81-9439-0fef57db1aa0/volumes" Nov 04 01:23:43.718358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:43.717835 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:23:44.678513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:44.678478 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:23:44.679241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:44.679214 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:23:46.800278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.800240 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh"] Nov 04 01:23:46.801579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801551 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" Nov 04 01:23:46.801579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801577 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801596 4312 cpu_manager.go:414] "containerMap: removing container" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801605 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801655 4312 cpu_manager.go:414] "containerMap: removing container" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801663 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801683 4312 cpu_manager.go:414] "containerMap: removing container" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801691 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801726 4312 cpu_manager.go:414] "containerMap: removing container" podUID="faa96b9c-c094-4d25-b5e0-de31751312e3" containerName="payload-pre-processing" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801733 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa96b9c-c094-4d25-b5e0-de31751312e3" containerName="payload-pre-processing" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801753 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1b880e25-057e-4833-9dcb-a2c529abeb64" containerName="payload-processing" Nov 04 01:23:46.801756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801759 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b880e25-057e-4833-9dcb-a2c529abeb64" containerName="payload-processing" Nov 04 01:23:46.802225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801934 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa96b9c-c094-4d25-b5e0-de31751312e3" containerName="payload-pre-processing" Nov 04 01:23:46.802225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801950 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b880e25-057e-4833-9dcb-a2c529abeb64" containerName="payload-processing" Nov 04 01:23:46.802225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801968 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1355eaa-1dac-49d2-9dab-31ba22187e0e" containerName="istio-proxy" Nov 04 01:23:46.802225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801979 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="551f2bc6-f790-4c9b-af7a-c183240e3b68" containerName="istio-proxy" Nov 04 01:23:46.802225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.801997 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="af31fac3-3297-4d81-9439-0fef57db1aa0" containerName="istio-proxy" Nov 04 01:23:46.802225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.802006 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="836f5631-21d0-4b71-99cd-1d0a82c8a855" containerName="istio-proxy" Nov 04 01:23:46.808568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.808543 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.817633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.817537 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-20a250-serving-cert\"" Nov 04 01:23:46.823098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.821020 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh"] Nov 04 01:23:46.841056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.841028 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-metrics-certs\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.841191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.841077 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-openshift-service-ca\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.841191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.841113 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bv9h\" (UniqueName: \"kubernetes.io/projected/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-kube-api-access-2bv9h\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.841282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.841248 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.952358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.946501 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-openshift-service-ca\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.952358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.946566 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bv9h\" (UniqueName: \"kubernetes.io/projected/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-kube-api-access-2bv9h\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.952358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.946705 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.952358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.947055 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-metrics-certs\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.963356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.960691 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-metrics-certs\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.963356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:46.961383 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-openshift-service-ca\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:46.963356 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:46.961786 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-serving-cert: secret "maas-api-e2e-api-b-20a250-serving-cert" not found Nov 04 01:23:46.963356 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:46.961851 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls podName:9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3 nodeName:}" failed. No retries permitted until 2026-11-04 01:23:47.461830959 +0000 UTC m=+1259.484762950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls") pod "maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" (UID: "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3") : secret "maas-api-e2e-api-b-20a250-serving-cert" not found Nov 04 01:23:47.008352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:47.008150 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bv9h\" (UniqueName: \"kubernetes.io/projected/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-kube-api-access-2bv9h\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:47.561600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:47.561568 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:47.563998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:47.563973 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls\") pod \"maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:47.745357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:47.740820 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:48.093937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.093863 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754"] Nov 04 01:23:48.117126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.117098 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.122724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.122698 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fdd516-serving-cert\"" Nov 04 01:23:48.162832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.162794 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754"] Nov 04 01:23:48.170530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.169707 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.170530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.169869 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.170530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.169909 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftg5k\" (UniqueName: \"kubernetes.io/projected/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-kube-api-access-ftg5k\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.170530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.170030 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-metrics-certs\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.245377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.240099 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh"] Nov 04 01:23:48.245377 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:23:48.242637 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac6e4c7_aa62_4013_afe7_3e4c0e5e1fe3.slice/crio-164835ce060dd34aab14b1ffa7c33b551ab69bcc80b2bee465e1a77a4804076a WatchSource:0}: Error finding container 164835ce060dd34aab14b1ffa7c33b551ab69bcc80b2bee465e1a77a4804076a: Status 404 returned error can't find the container with id 164835ce060dd34aab14b1ffa7c33b551ab69bcc80b2bee465e1a77a4804076a Nov 04 01:23:48.284361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.278243 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.297274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.297239 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.298796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.298759 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.303245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.303218 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftg5k\" (UniqueName: \"kubernetes.io/projected/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-kube-api-access-ftg5k\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.303873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.303830 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-metrics-certs\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.377475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.375952 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.378665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.378438 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftg5k\" (UniqueName: \"kubernetes.io/projected/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-kube-api-access-ftg5k\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.406375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.402025 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-metrics-certs\") pod \"maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.465468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.465433 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:48.802208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:48.802170 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754"] Nov 04 01:23:49.379357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:49.377914 4312 scope.go:122] "RemoveContainer" containerID="6f05d1c5bf6ce7c1d150e9ed57fdfbe7d68c7bf61b03a77b9f596b308db55d4a" Nov 04 01:23:49.404357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:49.404288 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" event={"ID":"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3","Type":"ContainerStarted","Data":"93eba80d40df3ee9bca1adb7bb70154a91a9bd78cbdc56d52fc1ab3a9c8c2418"} Nov 04 01:23:49.404523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:49.404383 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" event={"ID":"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3","Type":"ContainerStarted","Data":"164835ce060dd34aab14b1ffa7c33b551ab69bcc80b2bee465e1a77a4804076a"} Nov 04 01:23:49.404984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:49.404964 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:49.510357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:49.505570 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" event={"ID":"4304bb0b-2e6b-42c8-abaa-6d13d62519dc","Type":"ContainerStarted","Data":"f06d6f62c5498e5f7d1dde5054186985211e960c85b6bee23c27684e454559ee"} Nov 04 01:23:49.520210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:49.519882 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" podStartSLOduration=3.082268993 podStartE2EDuration="3.519862841s" totalImagesPullingTime="437.593848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:49.460094777 +0000 UTC m=+1261.483026782" watchObservedRunningTime="2026-11-04 01:23:49.519862841 +0000 UTC m=+1261.542794849" Nov 04 01:23:50.515344 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:50.515294 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" event={"ID":"4304bb0b-2e6b-42c8-abaa-6d13d62519dc","Type":"ContainerStarted","Data":"2a8a8d0b167b5974a794ee6fbc61ea9c386bd4cd7ff93f9da5097033b0f5e10e"} Nov 04 01:23:50.539303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:50.539252 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" podStartSLOduration=1.849683849 podStartE2EDuration="2.53923675s" totalImagesPullingTime="689.552901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:50.535530235 +0000 UTC m=+1262.558462262" watchObservedRunningTime="2026-11-04 01:23:50.53923675 +0000 UTC m=+1262.562168757" Nov 04 01:23:51.524778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:51.524742 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:51.542264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:51.542227 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7"] Nov 04 01:23:51.542513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:51.542484 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" podUID="1ae3bde8-b116-422a-92a7-2e23f76a71b2" containerName="maas-api" containerID="cri-o://503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae" gracePeriod=30 Nov 04 01:23:52.351833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.351804 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:23:52.381531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.381494 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-metrics-certs\") pod \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " Nov 04 01:23:52.385732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.385700 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-metrics-certs" pod "1ae3bde8-b116-422a-92a7-2e23f76a71b2" (UID: "1ae3bde8-b116-422a-92a7-2e23f76a71b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:52.482947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.482917 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae3bde8-b116-422a-92a7-2e23f76a71b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae3bde8-b116-422a-92a7-2e23f76a71b2-openshift-service-ca\") pod \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " Nov 04 01:23:52.483183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.483166 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-maas-api-tls\") pod \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " Nov 04 01:23:52.483271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.483199 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae3bde8-b116-422a-92a7-2e23f76a71b2-kube-api-access-csxs2\" (UniqueName: \"kubernetes.io/projected/1ae3bde8-b116-422a-92a7-2e23f76a71b2-kube-api-access-csxs2\") pod \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\" (UID: \"1ae3bde8-b116-422a-92a7-2e23f76a71b2\") " Nov 04 01:23:52.483379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.483317 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae3bde8-b116-422a-92a7-2e23f76a71b2-openshift-service-ca" pod "1ae3bde8-b116-422a-92a7-2e23f76a71b2" (UID: "1ae3bde8-b116-422a-92a7-2e23f76a71b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:52.483876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.483859 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:52.483938 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.483878 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae3bde8-b116-422a-92a7-2e23f76a71b2-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:52.485403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.485382 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-maas-api-tls" pod "1ae3bde8-b116-422a-92a7-2e23f76a71b2" (UID: "1ae3bde8-b116-422a-92a7-2e23f76a71b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:52.485478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.485406 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae3bde8-b116-422a-92a7-2e23f76a71b2-kube-api-access-csxs2" pod "1ae3bde8-b116-422a-92a7-2e23f76a71b2" (UID: "1ae3bde8-b116-422a-92a7-2e23f76a71b2"). InnerVolumeSpecName "kube-api-access-csxs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:52.533464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.533437 4312 generic.go:356] "Generic (PLEG): container finished" podID="1ae3bde8-b116-422a-92a7-2e23f76a71b2" containerID="503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae" exitCode=0 Nov 04 01:23:52.533873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.533478 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" Nov 04 01:23:52.533873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.533523 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" event={"ID":"1ae3bde8-b116-422a-92a7-2e23f76a71b2","Type":"ContainerDied","Data":"503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae"} Nov 04 01:23:52.533873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.533586 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7" event={"ID":"1ae3bde8-b116-422a-92a7-2e23f76a71b2","Type":"ContainerDied","Data":"705f8805edbcfb4000d683954bb8df36f2ce5b527cfbe24bc1df22901f8ae1c0"} Nov 04 01:23:52.533873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.533621 4312 scope.go:122] "RemoveContainer" containerID="503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae" Nov 04 01:23:52.544611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.544445 4312 scope.go:122] "RemoveContainer" containerID="503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae" Nov 04 01:23:52.544736 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:52.544718 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae\": container with ID starting with 503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae not found: ID does not exist" containerID="503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae" Nov 04 01:23:52.544783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.544743 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae"} err="failed to get container status \"503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae\": rpc error: code = NotFound desc = could not find container \"503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae\": container with ID starting with 503a473774d8e3a46837581e0fc1f38b7d190a5f1c014a839e067abec71340ae not found: ID does not exist" Nov 04 01:23:52.559875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.559850 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7"] Nov 04 01:23:52.564498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.564466 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7742169-6d555879d7-jlpd7"] Nov 04 01:23:52.586680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.586284 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae3bde8-b116-422a-92a7-2e23f76a71b2-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:52.586680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:52.586328 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csxs2\" (UniqueName: \"kubernetes.io/projected/1ae3bde8-b116-422a-92a7-2e23f76a71b2-kube-api-access-csxs2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:54.112362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.112307 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae3bde8-b116-422a-92a7-2e23f76a71b2" path="/var/lib/kubelet/pods/1ae3bde8-b116-422a-92a7-2e23f76a71b2/volumes" Nov 04 01:23:54.779674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.779643 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2"] Nov 04 01:23:54.780579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.780555 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1ae3bde8-b116-422a-92a7-2e23f76a71b2" containerName="maas-api" Nov 04 01:23:54.780579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.780576 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae3bde8-b116-422a-92a7-2e23f76a71b2" containerName="maas-api" Nov 04 01:23:54.780771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.780719 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae3bde8-b116-422a-92a7-2e23f76a71b2" containerName="maas-api" Nov 04 01:23:54.787434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.787411 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.802923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.802869 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2"] Nov 04 01:23:54.832668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.832637 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.832843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.832761 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.832843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.832835 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.833089 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.833070 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.833129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.833108 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.833185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.833133 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.833236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.833222 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574rj\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-kube-api-access-574rj\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.833289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.833279 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.833388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.833304 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947083 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947171 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947206 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947259 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574rj\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-kube-api-access-574rj\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947373 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947409 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947481 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947600 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.947666 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.948715 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.950587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.949086 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.980302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.980068 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.981046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.981013 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.985347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.981514 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:54.993346 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:54.985840 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:55.020407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.013602 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:55.026364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.025867 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574rj\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-kube-api-access-574rj\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:55.032794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.032714 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-7rsc2\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:55.114047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.114019 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:55.408244 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:23:55.408213 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dd4b10_4202_4b7b_9759_2e7a9f8b9c5e.slice/crio-32e2305fecc43f2087df1d722cf17c38e27381305ed84fc8b026747ef8163498 WatchSource:0}: Error finding container 32e2305fecc43f2087df1d722cf17c38e27381305ed84fc8b026747ef8163498: Status 404 returned error can't find the container with id 32e2305fecc43f2087df1d722cf17c38e27381305ed84fc8b026747ef8163498 Nov 04 01:23:55.409208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.409179 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2"] Nov 04 01:23:55.411756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.411722 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:23:55.411865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.411798 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:23:55.411865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.411843 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:23:55.477015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.476989 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6"] Nov 04 01:23:55.502265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.502226 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb"] Nov 04 01:23:55.532152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.532126 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:23:55.567917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.567880 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" event={"ID":"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e","Type":"ContainerStarted","Data":"a75542098305c2347d93f520d4a090c0102febe9015b5804640881e0b107a92f"} Nov 04 01:23:55.568185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.568166 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" event={"ID":"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e","Type":"ContainerStarted","Data":"32e2305fecc43f2087df1d722cf17c38e27381305ed84fc8b026747ef8163498"} Nov 04 01:23:55.596472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:55.596415 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podStartSLOduration=1.596396354 podStartE2EDuration="1.596396354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:55.595845097 +0000 UTC m=+1267.618777143" watchObservedRunningTime="2026-11-04 01:23:55.596396354 +0000 UTC m=+1267.619328365" Nov 04 01:23:56.115155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:56.114526 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:23:56.121387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:56.119595 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:23:56.121387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:56.119672 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 04 01:23:57.114694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.114657 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:23:57.114872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.114708 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 04 01:23:57.437047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.436970 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv"] Nov 04 01:23:57.442307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.442288 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.447209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.446830 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6e609c-openshift-default-dockercfg-n2474\"" Nov 04 01:23:57.484447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.484414 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv"] Nov 04 01:23:57.539561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.539534 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:23:57.588047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588007 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588075 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588235 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2b9\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-kube-api-access-jd2b9\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588268 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588448 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588537 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588617 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588666 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.588781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.588766 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.695807 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.695923 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696024 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696094 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696211 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2b9\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-kube-api-access-jd2b9\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696243 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696277 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.696482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696438 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.697024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696509 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.697024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.696593 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.697858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.697831 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.698126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.698109 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.698513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.698492 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.698676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.698535 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.699907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.699882 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.700141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.700124 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.708266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.708245 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.709292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.709269 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2b9\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-kube-api-access-jd2b9\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:57.791773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:57.791747 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:58.112148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.112122 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv"] Nov 04 01:23:58.115497 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.115465 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:23:58.115673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.115520 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 04 01:23:58.118316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.118283 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:23:58.118442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.118379 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:23:58.118442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.118425 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:23:58.599549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.599513 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" event={"ID":"4347a44e-f700-4c34-8bb8-4f877ddaf5a2","Type":"ContainerStarted","Data":"b16842c1e0e285c4ffc53b6b8fc595248f7d2698a1af04bf24348b3a454f410b"} Nov 04 01:23:58.599948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.599571 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" event={"ID":"4347a44e-f700-4c34-8bb8-4f877ddaf5a2","Type":"ContainerStarted","Data":"3eb7cfd16e53e6c88557362536835fd353c77e514e9e0f95edfb5f81d01ae001"} Nov 04 01:23:58.627067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.627006 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podStartSLOduration=1.62698799 podStartE2EDuration="1.62698799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:58.621652434 +0000 UTC m=+1270.644584466" watchObservedRunningTime="2026-11-04 01:23:58.62698799 +0000 UTC m=+1270.649919998" Nov 04 01:23:58.792095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:58.792062 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:23:59.114388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:59.114351 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:23:59.114584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:59.114408 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 04 01:23:59.481473 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:23:59.481393 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:23:59.793975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:59.793935 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:23:59.794362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:23:59.793987 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 04 01:24:00.114404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.114292 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:24:00.114563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.114396 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 04 01:24:00.492415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.491449 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" podUID="a5351dda-dd30-40fc-b844-c06fb755d763" containerName="payload-processing" containerID="cri-o://847b8540eddf14b7f62df4f895426b8edbc65f7f0050307cb670ee2420121e20" gracePeriod=25 Nov 04 01:24:00.505527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.505470 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" podUID="e12f6c86-a8e5-4421-9b01-b85bae80d3da" containerName="payload-pre-processing" containerID="cri-o://bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d" gracePeriod=25 Nov 04 01:24:00.620088 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.620057 4312 generic.go:356] "Generic (PLEG): container finished" podID="a5351dda-dd30-40fc-b844-c06fb755d763" containerID="847b8540eddf14b7f62df4f895426b8edbc65f7f0050307cb670ee2420121e20" exitCode=0 Nov 04 01:24:00.620989 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.620960 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" event={"ID":"a5351dda-dd30-40fc-b844-c06fb755d763","Type":"ContainerDied","Data":"847b8540eddf14b7f62df4f895426b8edbc65f7f0050307cb670ee2420121e20"} Nov 04 01:24:00.811859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.811824 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:24:00.812434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:00.812401 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 04 01:24:01.039837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.030283 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:24:01.118673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.118586 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:24:01.130468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.130431 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 04 01:24:01.130982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.120808 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5351dda-dd30-40fc-b844-c06fb755d763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5351dda-dd30-40fc-b844-c06fb755d763-plugins-config-volume\") pod \"a5351dda-dd30-40fc-b844-c06fb755d763\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " Nov 04 01:24:01.131264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.131246 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5351dda-dd30-40fc-b844-c06fb755d763-kube-api-access-57wln\" (UniqueName: \"kubernetes.io/projected/a5351dda-dd30-40fc-b844-c06fb755d763-kube-api-access-57wln\") pod \"a5351dda-dd30-40fc-b844-c06fb755d763\" (UID: \"a5351dda-dd30-40fc-b844-c06fb755d763\") " Nov 04 01:24:01.132450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.132424 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5351dda-dd30-40fc-b844-c06fb755d763-plugins-config-volume" pod "a5351dda-dd30-40fc-b844-c06fb755d763" (UID: "a5351dda-dd30-40fc-b844-c06fb755d763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:01.139782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.139752 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5351dda-dd30-40fc-b844-c06fb755d763-kube-api-access-57wln" pod "a5351dda-dd30-40fc-b844-c06fb755d763" (UID: "a5351dda-dd30-40fc-b844-c06fb755d763"). InnerVolumeSpecName "kube-api-access-57wln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:01.234012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.233981 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:24:01.235471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.235436 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wln\" (UniqueName: \"kubernetes.io/projected/a5351dda-dd30-40fc-b844-c06fb755d763-kube-api-access-57wln\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:01.235471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.235473 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5351dda-dd30-40fc-b844-c06fb755d763-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:01.336574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.336537 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12f6c86-a8e5-4421-9b01-b85bae80d3da-kube-api-access-gw7c7\" (UniqueName: \"kubernetes.io/projected/e12f6c86-a8e5-4421-9b01-b85bae80d3da-kube-api-access-gw7c7\") pod \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " Nov 04 01:24:01.336762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.336723 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12f6c86-a8e5-4421-9b01-b85bae80d3da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12f6c86-a8e5-4421-9b01-b85bae80d3da-plugins-config-volume\") pod \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\" (UID: \"e12f6c86-a8e5-4421-9b01-b85bae80d3da\") " Nov 04 01:24:01.337783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.337752 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12f6c86-a8e5-4421-9b01-b85bae80d3da-plugins-config-volume" pod "e12f6c86-a8e5-4421-9b01-b85bae80d3da" (UID: "e12f6c86-a8e5-4421-9b01-b85bae80d3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:01.339513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.339485 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12f6c86-a8e5-4421-9b01-b85bae80d3da-kube-api-access-gw7c7" pod "e12f6c86-a8e5-4421-9b01-b85bae80d3da" (UID: "e12f6c86-a8e5-4421-9b01-b85bae80d3da"). InnerVolumeSpecName "kube-api-access-gw7c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:01.440128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.440051 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7c7\" (UniqueName: \"kubernetes.io/projected/e12f6c86-a8e5-4421-9b01-b85bae80d3da-kube-api-access-gw7c7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:01.440128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.440076 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12f6c86-a8e5-4421-9b01-b85bae80d3da-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:01.629194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.629162 4312 generic.go:356] "Generic (PLEG): container finished" podID="e12f6c86-a8e5-4421-9b01-b85bae80d3da" containerID="bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d" exitCode=0 Nov 04 01:24:01.629389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.629203 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" Nov 04 01:24:01.629389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.629249 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" event={"ID":"e12f6c86-a8e5-4421-9b01-b85bae80d3da","Type":"ContainerDied","Data":"bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d"} Nov 04 01:24:01.629389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.629314 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb" event={"ID":"e12f6c86-a8e5-4421-9b01-b85bae80d3da","Type":"ContainerDied","Data":"bd25ca7b00be48823372ff6b346fd6bd3226f5ad7db3d1ae7d91ebed75a18629"} Nov 04 01:24:01.629389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.629368 4312 scope.go:122] "RemoveContainer" containerID="bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d" Nov 04 01:24:01.631711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.631691 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" event={"ID":"a5351dda-dd30-40fc-b844-c06fb755d763","Type":"ContainerDied","Data":"92fd3ff879177a592f1e44eda0cfbe55eb67384f8905dfdfce2cdc516ee83663"} Nov 04 01:24:01.631953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.631867 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6" Nov 04 01:24:01.642698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.642676 4312 scope.go:122] "RemoveContainer" containerID="bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d" Nov 04 01:24:01.642986 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:24:01.642959 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d\": container with ID starting with bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d not found: ID does not exist" containerID="bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d" Nov 04 01:24:01.643199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.642997 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d"} err="failed to get container status \"bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d\": rpc error: code = NotFound desc = could not find container \"bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d\": container with ID starting with bfb4c05ac1a6ad5755cc6e6792161148df70bc3a6df024e79867b27b2c46997d not found: ID does not exist" Nov 04 01:24:01.643199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.643020 4312 scope.go:122] "RemoveContainer" containerID="847b8540eddf14b7f62df4f895426b8edbc65f7f0050307cb670ee2420121e20" Nov 04 01:24:01.705587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.704153 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb"] Nov 04 01:24:01.708952 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.708926 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27bc4f8b-7cfd77b5bc-kt5kb"] Nov 04 01:24:01.713278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.713266 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6"] Nov 04 01:24:01.717012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.716993 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27bc4f8b-64fbc89848-ff2z6"] Nov 04 01:24:01.799360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.797811 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:24:01.799360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:01.797906 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 04 01:24:02.116406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:02.114511 4312 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:24:02.116406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:02.114573 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 04 01:24:02.116406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:02.115101 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5351dda-dd30-40fc-b844-c06fb755d763" path="/var/lib/kubelet/pods/a5351dda-dd30-40fc-b844-c06fb755d763/volumes" Nov 04 01:24:02.116406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:02.115683 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12f6c86-a8e5-4421-9b01-b85bae80d3da" path="/var/lib/kubelet/pods/e12f6c86-a8e5-4421-9b01-b85bae80d3da/volumes" Nov 04 01:24:02.792630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:02.792593 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:24:02.792871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:02.792654 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 04 01:24:03.119876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:03.119797 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:24:03.120358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:03.120322 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:24:03.121412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:03.121379 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:24:03.792268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:03.792233 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:24:03.792523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:03.792286 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 04 01:24:04.792902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:04.792872 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:24:04.793283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:04.792924 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 04 01:24:05.792763 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:05.792722 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:24:05.792961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:05.792780 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 04 01:24:06.147727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:06.147639 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd"] Nov 04 01:24:06.147903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:06.147808 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" containerID="cri-o://79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc" gracePeriod=30 Nov 04 01:24:06.797571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:06.797541 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:24:06.798047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:06.798033 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:24:06.798644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:06.798621 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:24:10.378802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:10.378753 4312 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:24:10.379282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:10.378818 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:24:11.461613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.461585 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:24:11.583875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.583840 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-podinfo\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.583910 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-token\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-token\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.583956 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-socket\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584212 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584191 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-kube-api-access-9h92d\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584231 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-data\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584313 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-socket" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:24:11.584441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584385 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-certs\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584487 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-envoy\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584540 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-data" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:24:11.584591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584560 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-credential-socket\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584728 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584711 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e128a83-5ec3-4618-b5bf-b32324dc1270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e128a83-5ec3-4618-b5bf-b32324dc1270-istiod-ca-cert\") pod \"0e128a83-5ec3-4618-b5bf-b32324dc1270\" (UID: \"0e128a83-5ec3-4618-b5bf-b32324dc1270\") " Nov 04 01:24:11.584784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584704 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-certs" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:24:11.584923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.584874 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-credential-socket" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:24:11.585124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.585101 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e128a83-5ec3-4618-b5bf-b32324dc1270-istiod-ca-cert" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:11.585633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.585612 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e128a83-5ec3-4618-b5bf-b32324dc1270-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.585695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.585640 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.585695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.585654 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.585695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.585668 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.585695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.585683 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.586432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.586405 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-token" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:11.586511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.586489 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-kube-api-access-9h92d" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "kube-api-access-9h92d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:11.586609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.586587 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-podinfo" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:24:11.586802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.586781 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-envoy" pod "0e128a83-5ec3-4618-b5bf-b32324dc1270" (UID: "0e128a83-5ec3-4618-b5bf-b32324dc1270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:24:11.687505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.687413 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-kube-api-access-9h92d\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.687505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.687442 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.687505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.687452 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.687505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.687462 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e128a83-5ec3-4618-b5bf-b32324dc1270-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:11.736814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.736783 4312 generic.go:356] "Generic (PLEG): container finished" podID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerID="79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc" exitCode=0 Nov 04 01:24:11.736986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.736830 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" Nov 04 01:24:11.736986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.736866 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" event={"ID":"0e128a83-5ec3-4618-b5bf-b32324dc1270","Type":"ContainerDied","Data":"79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc"} Nov 04 01:24:11.736986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.736945 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd" event={"ID":"0e128a83-5ec3-4618-b5bf-b32324dc1270","Type":"ContainerDied","Data":"c4db4cb99f0d5262b00157b61eb6b7f377c359f849044423bb9d5448fdab1f05"} Nov 04 01:24:11.736986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.736967 4312 scope.go:122] "RemoveContainer" containerID="79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc" Nov 04 01:24:11.750007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.749988 4312 scope.go:122] "RemoveContainer" containerID="79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc" Nov 04 01:24:11.750308 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:24:11.750281 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc\": container with ID starting with 79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc not found: ID does not exist" containerID="79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc" Nov 04 01:24:11.750444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.750319 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc"} err="failed to get container status \"79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc\": rpc error: code = NotFound desc = could not find container \"79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc\": container with ID starting with 79b5f71dacff9a22bc2b26f622295759d8456b34113e34154109d84168f51cfc not found: ID does not exist" Nov 04 01:24:11.765080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.765058 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd"] Nov 04 01:24:11.769571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:11.769551 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-77vkd"] Nov 04 01:24:12.112150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:12.112116 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" path="/var/lib/kubelet/pods/0e128a83-5ec3-4618-b5bf-b32324dc1270/volumes" Nov 04 01:24:14.826373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.826324 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d"] Nov 04 01:24:14.827281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827258 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" Nov 04 01:24:14.827281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827278 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" Nov 04 01:24:14.827467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827327 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a5351dda-dd30-40fc-b844-c06fb755d763" containerName="payload-processing" Nov 04 01:24:14.827467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827347 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5351dda-dd30-40fc-b844-c06fb755d763" containerName="payload-processing" Nov 04 01:24:14.827467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827367 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e12f6c86-a8e5-4421-9b01-b85bae80d3da" containerName="payload-pre-processing" Nov 04 01:24:14.827467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827373 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12f6c86-a8e5-4421-9b01-b85bae80d3da" containerName="payload-pre-processing" Nov 04 01:24:14.827625 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827508 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e128a83-5ec3-4618-b5bf-b32324dc1270" containerName="istio-proxy" Nov 04 01:24:14.827625 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827522 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5351dda-dd30-40fc-b844-c06fb755d763" containerName="payload-processing" Nov 04 01:24:14.827625 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.827531 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12f6c86-a8e5-4421-9b01-b85bae80d3da" containerName="payload-pre-processing" Nov 04 01:24:14.832845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.832828 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.844791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.844767 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d"] Nov 04 01:24:14.939654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.939621 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.939834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.939659 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cws2m\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-kube-api-access-cws2m\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.939834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.939793 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.939834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.939830 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.939991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.939861 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.940061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.940041 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.940103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.940089 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.940251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.940238 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:14.940300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:14.940286 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.041793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.041762 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.041949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.041809 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.041949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.041920 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.041963 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042019 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042112 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042049 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws2m\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-kube-api-access-cws2m\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042104 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042132 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042161 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042169 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042227 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042436 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042537 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.042731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.042708 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.044231 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.044206 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.044596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.044578 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.054560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.054538 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.054660 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.054622 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cws2m\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-kube-api-access-cws2m\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.149214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.149143 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:15.440469 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:24:15.440427 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecd9d73_fcf8_4b10_9ed4_02edf428baee.slice/crio-d14458bfd26684d2cf76bbd23f2f06362dfb4c84d1d36d7c3e467dfb72f683a7 WatchSource:0}: Error finding container d14458bfd26684d2cf76bbd23f2f06362dfb4c84d1d36d7c3e467dfb72f683a7: Status 404 returned error can't find the container with id d14458bfd26684d2cf76bbd23f2f06362dfb4c84d1d36d7c3e467dfb72f683a7 Nov 04 01:24:15.441647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.441547 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d"] Nov 04 01:24:15.444220 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.444181 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:15.444321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.444264 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:15.444321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.444307 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:15.771288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.771195 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" event={"ID":"9ecd9d73-fcf8-4b10-9ed4-02edf428baee","Type":"ContainerStarted","Data":"034b430f98e3680852a0768fbe012b964fff860e691c7ce442d5f7aac13f3d0b"} Nov 04 01:24:15.771288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.771264 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" event={"ID":"9ecd9d73-fcf8-4b10-9ed4-02edf428baee","Type":"ContainerStarted","Data":"d14458bfd26684d2cf76bbd23f2f06362dfb4c84d1d36d7c3e467dfb72f683a7"} Nov 04 01:24:15.796719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:15.796670 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podStartSLOduration=1.79665604 podStartE2EDuration="1.79665604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:15.794614737 +0000 UTC m=+1287.817546761" watchObservedRunningTime="2026-11-04 01:24:15.79665604 +0000 UTC m=+1287.819588046" Nov 04 01:24:16.150040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:16.149991 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:16.151560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:16.151535 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:16.151695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:16.151590 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:17.149694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:17.149660 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:17.149875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:17.149717 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:18.149861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:18.149830 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:18.150273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:18.149878 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:19.149984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:19.149953 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:19.150396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:19.150007 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:20.150395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:20.150361 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:20.150791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:20.150411 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:21.150311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:21.150267 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:21.150531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:21.150324 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:22.149736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:22.149705 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:22.149896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:22.149758 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:23.150127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:23.150094 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:23.150539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:23.150144 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:24.149621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:24.149587 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:24.149811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:24.149642 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:25.150121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:25.150086 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:25.150695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:25.150152 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:26.150283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.150248 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:26.150713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.150313 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:26.778423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.778386 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf"] Nov 04 01:24:26.786522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.786496 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.791511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.791062 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d2e4e3-openshift-default-dockercfg-9wb7h\"" Nov 04 01:24:26.803009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.802973 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf"] Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937265 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-certs\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937469 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hhj\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-kube-api-access-d7hhj\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937567 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-socket\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937607 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-data\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937679 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-istio-token\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937797 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc63742-f448-4195-bbe2-d15647027916-istiod-ca-cert\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-credential-socket\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937904 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc63742-f448-4195-bbe2-d15647027916-istio-podinfo\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:26.939358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:26.937931 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-envoy\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039429 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hhj\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-kube-api-access-d7hhj\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039539 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-socket\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039583 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-data\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039659 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-istio-token\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039831 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc63742-f448-4195-bbe2-d15647027916-istiod-ca-cert\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039866 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-credential-socket\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039941 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-socket\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039959 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-data\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.039992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.039965 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc63742-f448-4195-bbe2-d15647027916-istio-podinfo\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.040153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.040008 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-envoy\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.040153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.040044 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-certs\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.040259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.040230 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-credential-socket\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.040432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.040407 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-certs\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.040622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.040598 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc63742-f448-4195-bbe2-d15647027916-istiod-ca-cert\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.042347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.042309 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-envoy\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.042486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.042464 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc63742-f448-4195-bbe2-d15647027916-istio-podinfo\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.051686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.051660 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-istio-token\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.051823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.051725 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hhj\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-kube-api-access-d7hhj\") pod \"e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.129360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.129309 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:27.157174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.154682 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:27.157174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.154740 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:27.444841 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:24:27.444809 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc63742_f448_4195_bbe2_d15647027916.slice/crio-fd4a44635982220716ab7940abb0c690d6c6454a3e243a3a5559a4f02aa2090d WatchSource:0}: Error finding container fd4a44635982220716ab7940abb0c690d6c6454a3e243a3a5559a4f02aa2090d: Status 404 returned error can't find the container with id fd4a44635982220716ab7940abb0c690d6c6454a3e243a3a5559a4f02aa2090d Nov 04 01:24:27.446554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.446523 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf"] Nov 04 01:24:27.448438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.448400 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:27.448539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.448473 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:27.448539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.448509 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:27.962382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.962345 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerStarted","Data":"130915f29c020dc2ed76bf5521f8e38f88aa2c989e86586cfba75df8d64347cb"} Nov 04 01:24:27.962598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.962403 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerStarted","Data":"fd4a44635982220716ab7940abb0c690d6c6454a3e243a3a5559a4f02aa2090d"} Nov 04 01:24:27.990200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:27.990148 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podStartSLOduration=1.990133241 podStartE2EDuration="1.990133241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:27.984965148 +0000 UTC m=+1300.007897156" watchObservedRunningTime="2026-11-04 01:24:27.990133241 +0000 UTC m=+1300.013065305" Nov 04 01:24:28.150022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:28.149980 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:28.156369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:28.152980 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:28.156369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:28.153035 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:28.168697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:28.150281 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:28.181359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:28.173652 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:29.130576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:29.130488 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:29.130576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:29.130543 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:29.150122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:29.150084 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:29.150279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:29.150136 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:29.460470 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:24:29.460376 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:24:30.130790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:30.130755 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:30.130941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:30.130818 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:30.149725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:30.149700 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:30.149906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:30.149885 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:30.149965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:30.149927 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:31.130195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:31.130158 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:31.130669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:31.130221 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:31.150158 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:31.150126 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:31.150295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:31.150175 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:32.131101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:32.131065 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:32.131707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:32.131135 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:32.149471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:32.149442 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:32.149624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:32.149500 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:33.131066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:33.131028 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:33.131276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:33.131083 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:33.150130 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:33.150098 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:33.150284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:33.150162 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:34.131368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:34.131303 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:34.131783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:34.131385 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:34.149567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:34.149536 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:34.149703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:34.149579 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:35.131253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:35.131220 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:35.131447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:35.131280 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:35.149524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:35.149485 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:35.149666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:35.149548 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:36.130816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:36.130784 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:36.131001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:36.130829 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:36.150315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:36.150284 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:36.150692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:36.150372 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:37.130519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:37.130480 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:37.130728 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:37.130554 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:37.149867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:37.149832 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:37.150032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:37.149892 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:38.048249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.048217 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9"] Nov 04 01:24:38.055637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.055602 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.060978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.060958 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-26f3d2-openshift-default-dockercfg-w9m8v\"" Nov 04 01:24:38.063986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.063965 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9"] Nov 04 01:24:38.131092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.131059 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:38.131265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.131117 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:38.148287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148252 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvdw\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-kube-api-access-9dvdw\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.148452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148329 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-podinfo\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.148508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148488 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istiod-ca-cert\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.148557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148531 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-credential-socket\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.148557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148550 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-socket\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.148766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148748 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-envoy\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.148924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148903 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-token\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.148975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.148937 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-certs\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.149051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.149029 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-data\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.149407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.149385 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:38.149515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.149435 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.256590 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-envoy\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.256765 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-token\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.256797 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-certs\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.256855 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-data\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.257030 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvdw\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-kube-api-access-9dvdw\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.257119 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-podinfo\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.257204 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istiod-ca-cert\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.257245 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-credential-socket\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.257276 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-socket\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.257814 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-socket\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.258971 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-certs\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.262359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.259202 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-data\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.281404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.279122 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istiod-ca-cert\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.281404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.280267 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-credential-socket\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.282896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.282844 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-envoy\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.288051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.288020 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvdw\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-kube-api-access-9dvdw\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.288583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.288529 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-podinfo\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.321967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.321809 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-token\") pod \"e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.375136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.375107 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:38.665679 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:24:38.665646 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593b2775_f312_4b74_9c40_c0ab0fdf5b7d.slice/crio-baeaa19a1fc299441ebfc846d6a95c23d4a79d2e228543871f8b86b1e12bc6a5 WatchSource:0}: Error finding container baeaa19a1fc299441ebfc846d6a95c23d4a79d2e228543871f8b86b1e12bc6a5: Status 404 returned error can't find the container with id baeaa19a1fc299441ebfc846d6a95c23d4a79d2e228543871f8b86b1e12bc6a5 Nov 04 01:24:38.666273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.666208 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9"] Nov 04 01:24:38.668343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.668296 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:38.668425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.668396 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:38.668466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:38.668432 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:39.061434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.061395 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" event={"ID":"593b2775-f312-4b74-9c40-c0ab0fdf5b7d","Type":"ContainerStarted","Data":"b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9"} Nov 04 01:24:39.061844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.061462 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" event={"ID":"593b2775-f312-4b74-9c40-c0ab0fdf5b7d","Type":"ContainerStarted","Data":"baeaa19a1fc299441ebfc846d6a95c23d4a79d2e228543871f8b86b1e12bc6a5"} Nov 04 01:24:39.085823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.085776 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podStartSLOduration=1.085759357 podStartE2EDuration="1.085759357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:39.08477275 +0000 UTC m=+1311.107704775" watchObservedRunningTime="2026-11-04 01:24:39.085759357 +0000 UTC m=+1311.108691364" Nov 04 01:24:39.131271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.131242 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:39.131443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.131292 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:39.150102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.150075 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:39.150243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.150123 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:39.375782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.375699 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:39.387517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.387301 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:39.387702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:39.387535 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:40.130277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:40.130238 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:40.130759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:40.130288 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:40.150140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:40.150109 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:40.150290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:40.150168 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:40.376264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:40.376232 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:40.376454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:40.376280 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:41.131251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:41.131223 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:41.131664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:41.131272 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:41.149439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:41.149408 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:41.149610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:41.149466 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:41.375630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:41.375594 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:41.375796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:41.375661 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:42.130239 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:42.130205 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:42.130474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:42.130272 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:42.131714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:42.131695 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:42.149468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:42.149438 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:42.149598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:42.149493 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:42.375401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:42.375371 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:42.375566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:42.375421 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:43.130849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:43.130817 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:43.131023 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:43.130869 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:43.149974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:43.149944 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:43.150455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:43.150002 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:43.376050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:43.376013 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:43.376192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:43.376066 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:44.131361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.131303 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:44.131549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.131378 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:44.150074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.150039 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:44.150509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.150080 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:44.376075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.376039 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:44.376238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.376088 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:44.622111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.622068 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk"] Nov 04 01:24:44.628102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.628078 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.645402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.645371 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-943d4a-openshift-default-dockercfg-jvm6m\"" Nov 04 01:24:44.658026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.657320 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk"] Nov 04 01:24:44.778921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.778879 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6250006e-8963-4565-91e4-6d09ceadf762-istio-podinfo\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.781919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.781879 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-istio-token\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.782135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.782119 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-socket\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.782242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.782229 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-credential-socket\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.782461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.782427 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-data\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.788111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.788082 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-certs\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.788286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.788266 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-envoy\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.799753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.799717 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6250006e-8963-4565-91e4-6d09ceadf762-istiod-ca-cert\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.800034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.800011 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjrkv\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-kube-api-access-fjrkv\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902083 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6250006e-8963-4565-91e4-6d09ceadf762-istio-podinfo\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902145 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-istio-token\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902171 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-socket\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902189 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-credential-socket\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902251 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-data\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902369 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-certs\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902395 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-envoy\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902564 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6250006e-8963-4565-91e4-6d09ceadf762-istiod-ca-cert\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902653 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrkv\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-kube-api-access-fjrkv\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902694 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-credential-socket\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902802 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-certs\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902895 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-socket\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.902956 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.902936 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-data\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.903310 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.903290 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6250006e-8963-4565-91e4-6d09ceadf762-istiod-ca-cert\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.904903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.904882 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6250006e-8963-4565-91e4-6d09ceadf762-istio-podinfo\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.904987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.904920 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-envoy\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.918378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.918351 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-istio-token\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.919540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.919517 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrkv\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-kube-api-access-fjrkv\") pod \"e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:44.963097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:44.963060 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:45.130031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.129991 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:45.130605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.130058 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:45.150402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.150354 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:45.150837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.150430 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:45.150837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.150477 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:45.151152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.151094 4312 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"034b430f98e3680852a0768fbe012b964fff860e691c7ce442d5f7aac13f3d0b"} pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:24:45.151211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.151156 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" containerID="cri-o://034b430f98e3680852a0768fbe012b964fff860e691c7ce442d5f7aac13f3d0b" gracePeriod=30 Nov 04 01:24:45.263531 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:24:45.263500 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6250006e_8963_4565_91e4_6d09ceadf762.slice/crio-86e73587fa99f44c836462c7333c7d9ac7cf331571b4efb1c65398bb5acbc66d WatchSource:0}: Error finding container 86e73587fa99f44c836462c7333c7d9ac7cf331571b4efb1c65398bb5acbc66d: Status 404 returned error can't find the container with id 86e73587fa99f44c836462c7333c7d9ac7cf331571b4efb1c65398bb5acbc66d Nov 04 01:24:45.263676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.263583 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk"] Nov 04 01:24:45.266416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.266386 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:45.266527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.266446 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:45.266527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.266472 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:45.375917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.375885 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:45.376045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:45.375932 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:46.130322 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.130285 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:46.130512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.130362 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:46.134215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.134187 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" event={"ID":"6250006e-8963-4565-91e4-6d09ceadf762","Type":"ContainerStarted","Data":"58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0"} Nov 04 01:24:46.134327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.134236 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" event={"ID":"6250006e-8963-4565-91e4-6d09ceadf762","Type":"ContainerStarted","Data":"86e73587fa99f44c836462c7333c7d9ac7cf331571b4efb1c65398bb5acbc66d"} Nov 04 01:24:46.158741 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.158699 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podStartSLOduration=2.158685011 podStartE2EDuration="2.158685011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:46.157317044 +0000 UTC m=+1318.180249081" watchObservedRunningTime="2026-11-04 01:24:46.158685011 +0000 UTC m=+1318.181617019" Nov 04 01:24:46.378283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.378241 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:46.378565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.378533 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:46.963860 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.963826 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:46.965499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.965472 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:46.965643 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:46.965522 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:47.060615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:47.060585 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cs7zr_dc46f559-06ba-4445-8048-f29d73b9f168/konnectivity-agent/0.log" limit=1024 Nov 04 01:24:47.130784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:47.130751 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:47.130949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:47.130798 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:47.375792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:47.375760 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:47.376160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:47.375817 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:47.963891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:47.963859 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:47.964060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:47.963920 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:48.130617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:48.130587 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:48.130773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:48.130632 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:48.376401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:48.376366 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:48.376861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:48.376431 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:48.963277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:48.963237 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:48.963483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:48.963291 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:49.130682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:49.130650 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:49.130844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:49.130705 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:49.375593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:49.375558 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:49.375779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:49.375607 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:49.964117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:49.964044 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:49.964117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:49.964092 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:50.130482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.130451 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:50.130650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.130501 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:50.265946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.265908 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:50.266035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.265983 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:50.266035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.266020 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:24:50.376365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.376314 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:50.376482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.376390 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:50.964410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.964377 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:50.964788 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:50.964431 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:51.131090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.131062 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:51.131254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.131111 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:51.175929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.175899 4312 generic.go:356] "Generic (PLEG): container finished" podID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerID="034b430f98e3680852a0768fbe012b964fff860e691c7ce442d5f7aac13f3d0b" exitCode=0 Nov 04 01:24:51.176095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.175958 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" event={"ID":"9ecd9d73-fcf8-4b10-9ed4-02edf428baee","Type":"ContainerDied","Data":"034b430f98e3680852a0768fbe012b964fff860e691c7ce442d5f7aac13f3d0b"} Nov 04 01:24:51.176095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.176022 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" event={"ID":"9ecd9d73-fcf8-4b10-9ed4-02edf428baee","Type":"ContainerStarted","Data":"07fdc2ae4a43cad95b88facbe9c90b77e3ddf606f66763256b31b52fabb0a3ae"} Nov 04 01:24:51.375859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.375823 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:51.376012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.375879 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:51.966748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.966706 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:51.967366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:51.967312 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:52.130848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.130819 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:52.131005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.130866 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:52.156545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.156509 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:52.157147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.157122 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:52.157324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.157303 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:52.375875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.375836 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:52.376059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.375894 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:52.969358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.968443 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:52.969358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:52.968502 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:53.131244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.131210 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:53.131511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.131260 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:53.166527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.166493 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:53.166805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.166777 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:53.375645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.375601 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:24:53.375867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.375845 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:53.375915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.375893 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:53.972769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.972729 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:53.973298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:53.972795 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:54.130321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.130289 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:54.130495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.130350 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:54.167123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.167088 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:54.167399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.167371 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:54.375634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.375591 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:54.375868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.375650 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:54.972362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.971433 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:54.972362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:54.971496 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:55.130489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.130451 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:55.130970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.130515 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:55.161362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.151656 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:55.161362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.151709 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:55.375978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.375941 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:55.376150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.375997 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:55.966359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.965694 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:55.966359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:55.965754 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:56.131499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.131468 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:56.131899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.131519 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:56.162397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.162364 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:24:56.162676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.162645 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:24:56.375883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.375842 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:56.376101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.375909 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:56.963419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.963384 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:56.963594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:56.963443 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:57.130264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.130234 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:24:57.130470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.130283 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:24:57.130470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.130317 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:24:57.130831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.130812 4312 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"130915f29c020dc2ed76bf5521f8e38f88aa2c989e86586cfba75df8d64347cb"} pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:24:57.130888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.130855 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" containerID="cri-o://130915f29c020dc2ed76bf5521f8e38f88aa2c989e86586cfba75df8d64347cb" gracePeriod=30 Nov 04 01:24:57.154632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.154605 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:57.155104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.155090 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:57.155725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.155706 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:24:57.376374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.376311 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:57.376551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.376381 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:57.964024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.963989 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:57.964199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:57.964039 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:58.376143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:58.376108 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:58.376577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:58.376164 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:58.963758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:58.963719 4312 patch_prober.go:29] interesting pod/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:24:58.963942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:58.963771 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:24:59.375410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:59.375373 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:24:59.375588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:59.375428 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:24:59.459118 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:24:59.459085 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:24:59.963970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:59.963938 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:24:59.969166 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:24:59.969140 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:25:00.248764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:00.248693 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:25:00.375916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:00.375879 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:25:00.376097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:00.375936 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:25:01.380178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:01.380142 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:25:01.381504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:01.381480 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:25:02.251189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:02.251153 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:25:02.251300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:02.251231 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:25:02.251300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:02.251264 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:25:02.265801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:02.265773 4312 generic.go:356] "Generic (PLEG): container finished" podID="abc63742-f448-4195-bbe2-d15647027916" containerID="130915f29c020dc2ed76bf5521f8e38f88aa2c989e86586cfba75df8d64347cb" exitCode=0 Nov 04 01:25:02.265909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:02.265839 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerDied","Data":"130915f29c020dc2ed76bf5521f8e38f88aa2c989e86586cfba75df8d64347cb"} Nov 04 01:25:03.276016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:03.275976 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerStarted","Data":"fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1"} Nov 04 01:25:04.130397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:04.130367 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:25:04.130577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:04.130552 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:04.130627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:04.130598 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:05.130639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:05.130601 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:05.131154 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:05.130666 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:06.130071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:06.130033 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:06.130250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:06.130092 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:07.130483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:07.130445 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:07.130950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:07.130504 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:08.130578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:08.130545 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:08.130968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:08.130601 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:09.130309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:09.130272 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:09.130529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:09.130357 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:10.130323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:10.130293 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:10.130800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:10.130385 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:11.131134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:11.131099 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:11.131628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:11.131149 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:12.130170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:12.130137 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:12.130366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:12.130183 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:12.132568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:12.132551 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:25:13.130415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:13.130385 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:13.130601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:13.130435 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:14.131677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:14.131640 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:14.132251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:14.131696 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:15.130664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:15.130626 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:15.130838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:15.130700 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:16.131447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:16.131416 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:16.131852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:16.131472 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:17.130059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:17.130029 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:17.130233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:17.130083 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:18.130118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:18.130084 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:18.130529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:18.130142 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:19.131570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:19.131531 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:19.131943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:19.131601 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:20.130451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:20.130415 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:20.130735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:20.130479 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:21.130573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:21.130540 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:21.131086 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:21.130601 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:21.243565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:21.243532 4312 ???:1] "http: TLS handshake error from 10.0.140.186:41560: read tcp 10.0.136.96:10250->10.0.140.186:41560: read: connection reset by peer" Nov 04 01:25:22.131731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:22.131701 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:25:22.132108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:22.131746 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:25:23.135430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:23.135399 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:25:23.136540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:23.136520 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:25:29.459269 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:25:29.459232 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:25:38.229238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:25:38.229205 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cs7zr_dc46f559-06ba-4445-8048-f29d73b9f168/konnectivity-agent/0.log" limit=1024 Nov 04 01:25:59.459165 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:25:59.459078 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:26:13.976167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.976131 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk"] Nov 04 01:26:13.976852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.976392 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" containerID="cri-o://2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b" gracePeriod=30 Nov 04 01:26:13.981982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.981955 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5"] Nov 04 01:26:13.982236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.982212 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" containerID="cri-o://7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf" gracePeriod=30 Nov 04 01:26:13.987396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.987372 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv"] Nov 04 01:26:13.987618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.987588 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" containerID="cri-o://543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6" gracePeriod=30 Nov 04 01:26:13.992815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.992792 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg"] Nov 04 01:26:13.993016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:13.992979 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" containerID="cri-o://265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c" gracePeriod=30 Nov 04 01:26:14.039425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.039391 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984"] Nov 04 01:26:14.039849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.039815 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" containerID="cri-o://2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623" gracePeriod=30 Nov 04 01:26:14.103236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.103182 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv"] Nov 04 01:26:14.105107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.104981 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" containerID="cri-o://b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128" gracePeriod=30 Nov 04 01:26:14.139602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.139557 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5"] Nov 04 01:26:14.139919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.139886 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb"] Nov 04 01:26:14.140216 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.140169 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" containerID="cri-o://8cd64a5f27260f300ff6a4454101d8a920120d7429ea5df4766d6bca392d3bcb" gracePeriod=30 Nov 04 01:26:14.140439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.140416 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" containerID="cri-o://9a8b7c1d965b0a15c1f29442e05227586381cecafa1f2aec9d492371b3fcf13f" gracePeriod=30 Nov 04 01:26:14.161258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.161228 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9"] Nov 04 01:26:14.161698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.161672 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" containerID="cri-o://6771ed881ea0585229b0e34a110ac47302f6233bbc1ebbde58f83926a45eb197" gracePeriod=30 Nov 04 01:26:14.170507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.170480 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929"] Nov 04 01:26:14.170723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.170689 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" containerID="cri-o://3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0" gracePeriod=30 Nov 04 01:26:14.177778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.177755 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf"] Nov 04 01:26:14.178157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.178111 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" containerID="cri-o://a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955" gracePeriod=30 Nov 04 01:26:14.190051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.190027 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r"] Nov 04 01:26:14.190244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.190225 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" containerID="cri-o://e524f96396a9fe01c9002bfd5d5cfd648009ffe3fcddd900ba674ab7beac4686" gracePeriod=30 Nov 04 01:26:14.510294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.510255 4312 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:26:14.510474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:14.510315 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:26:15.344879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:15.344841 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:26:15.345373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:15.344901 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:26:15.521531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:15.521494 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:26:15.521735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:15.521557 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:26:17.217482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:17.217447 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk"] Nov 04 01:26:17.220001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:17.218168 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" containerID="cri-o://58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0" gracePeriod=30 Nov 04 01:26:19.705268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.704586 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:26:19.722703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.708459 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:26:19.737701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.737626 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:26:19.750611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.750525 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:26:19.782358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.782253 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:26:19.818927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.818065 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:26:19.842633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.842535 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:26:19.853197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.853136 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-socket\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.853723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.853552 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-istio-token\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-istio-token\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.854157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854105 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-istio-token\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-istio-token\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854393 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8860371f-a537-41c7-ae99-7d5c165baed9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8860371f-a537-41c7-ae99-7d5c165baed9-istio-podinfo\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854434 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-data\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854472 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8860371f-a537-41c7-ae99-7d5c165baed9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8860371f-a537-41c7-ae99-7d5c165baed9-istiod-ca-cert\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854502 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-kube-api-access-9hzql\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-kube-api-access-9hzql\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854537 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/481d9419-e346-444c-b319-a38160b62de7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481d9419-e346-444c-b319-a38160b62de7-istio-podinfo\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854572 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-kube-api-access-5pmw4\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-kube-api-access-5pmw4\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854606 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-envoy\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854642 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-socket\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.854736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.854680 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-data\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.855436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855320 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-kube-api-access-w9j9z\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-kube-api-access-w9j9z\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855412 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-credential-socket\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855581 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-data\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855616 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-envoy\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855676 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istiod-ca-cert\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855711 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481d9419-e346-444c-b319-a38160b62de7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481d9419-e346-444c-b319-a38160b62de7-istiod-ca-cert\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855778 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-socket\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855820 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-credential-socket\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855900 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-credential-socket\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855967 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-socket\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.855995 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-data\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856038 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-token\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-token\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856080 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-certs\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856134 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-envoy\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856187 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-token\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-token\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856246 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-envoy\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856394 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-podinfo\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856425 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-certs\") pod \"8860371f-a537-41c7-ae99-7d5c165baed9\" (UID: \"8860371f-a537-41c7-ae99-7d5c165baed9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856456 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5168ab-db6a-423c-81e8-80011c9d13f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5168ab-db6a-423c-81e8-80011c9d13f9-istiod-ca-cert\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856498 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-podinfo\") pod \"dc5168ab-db6a-423c-81e8-80011c9d13f9\" (UID: \"dc5168ab-db6a-423c-81e8-80011c9d13f9\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856554 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-credential-socket\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856600 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-certs\") pod \"481d9419-e346-444c-b319-a38160b62de7\" (UID: \"481d9419-e346-444c-b319-a38160b62de7\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856633 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-kube-api-access-m2kc7\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-kube-api-access-m2kc7\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.856839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.856688 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-certs\") pod \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\" (UID: \"51f907ae-3f89-40f0-b5be-96fba4eff5c1\") " Nov 04 01:26:19.872446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.871184 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-socket" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.872446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.871445 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-socket" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.882195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.880862 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-envoy" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.882195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.881166 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-credential-socket" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.882195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.881403 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-credential-socket" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.882195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.881621 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-socket" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.882195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.881851 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-data" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.884510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.884455 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-credential-socket" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.912413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.912032 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-certs" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.915324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.915098 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-certs" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.915495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.915374 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-certs" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.917164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.917085 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.917164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.917123 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.917566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.917140 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.917725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.917683 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.917884 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.917841 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.918040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.917998 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.918198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.918153 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.918354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.918178 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.918354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.918321 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.918592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.918550 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.918751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.918711 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.919632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.919209 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5168ab-db6a-423c-81e8-80011c9d13f9-istiod-ca-cert" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:19.933883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.926981 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-socket" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.937992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.935964 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-data" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.937992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.936723 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481d9419-e346-444c-b319-a38160b62de7-istiod-ca-cert" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:19.940365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.938994 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-data" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.945173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.941806 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-credential-socket" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.945173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.942646 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istiod-ca-cert" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:19.945173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.942865 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-certs" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.946729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.946699 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8860371f-a537-41c7-ae99-7d5c165baed9-istiod-ca-cert" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:19.952385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.946971 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-data" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.970246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.970180 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:26:19.978886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.978469 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-kube-api-access-m2kc7" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "kube-api-access-m2kc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.978886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.978852 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-token" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.979195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.978957 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-kube-api-access-w9j9z" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "kube-api-access-w9j9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.979195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.979015 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-envoy" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.979416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.979389 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-kube-api-access-9hzql" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "kube-api-access-9hzql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.981826 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.980718 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-kube-api-access-5pmw4" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "kube-api-access-5pmw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.984766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.984292 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/481d9419-e346-444c-b319-a38160b62de7-istio-podinfo" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:19.990268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.987733 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-token" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.993137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.991422 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-istio-token" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.993137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.991572 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-podinfo" pod "51f907ae-3f89-40f0-b5be-96fba4eff5c1" (UID: "51f907ae-3f89-40f0-b5be-96fba4eff5c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:19.993137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.991650 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-envoy" pod "481d9419-e346-444c-b319-a38160b62de7" (UID: "481d9419-e346-444c-b319-a38160b62de7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.993137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.991819 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-istio-token" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.993137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.991968 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-envoy" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:19.995757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.995472 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8860371f-a537-41c7-ae99-7d5c165baed9-istio-podinfo" pod "8860371f-a537-41c7-ae99-7d5c165baed9" (UID: "8860371f-a537-41c7-ae99-7d5c165baed9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:19.995757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.995673 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-podinfo" pod "dc5168ab-db6a-423c-81e8-80011c9d13f9" (UID: "dc5168ab-db6a-423c-81e8-80011c9d13f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:19.995757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.995698 4312 generic.go:356] "Generic (PLEG): container finished" podID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerID="7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf" exitCode=0 Nov 04 01:26:19.995757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.995727 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" event={"ID":"dc5168ab-db6a-423c-81e8-80011c9d13f9","Type":"ContainerDied","Data":"7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf"} Nov 04 01:26:19.995974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.995776 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" Nov 04 01:26:19.995974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.995818 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5" event={"ID":"dc5168ab-db6a-423c-81e8-80011c9d13f9","Type":"ContainerDied","Data":"038613efe4eacfd4691f7608eb4a92d0fbaf0847b4ac3a93463e2a699a12aa91"} Nov 04 01:26:19.995974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.995852 4312 scope.go:122] "RemoveContainer" containerID="7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf" Nov 04 01:26:19.998290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.998246 4312 generic.go:356] "Generic (PLEG): container finished" podID="33dee613-4f76-4d13-9700-22a0d178c024" containerID="9a8b7c1d965b0a15c1f29442e05227586381cecafa1f2aec9d492371b3fcf13f" exitCode=0 Nov 04 01:26:19.998526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.998507 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" event={"ID":"33dee613-4f76-4d13-9700-22a0d178c024","Type":"ContainerDied","Data":"9a8b7c1d965b0a15c1f29442e05227586381cecafa1f2aec9d492371b3fcf13f"} Nov 04 01:26:19.999719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:19.999699 4312 generic.go:356] "Generic (PLEG): container finished" podID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerID="a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955" exitCode=0 Nov 04 01:26:20.000027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.000013 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" Nov 04 01:26:20.000179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.000165 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" event={"ID":"82a77e5e-42bb-4b2e-a233-942284a5e1e9","Type":"ContainerDied","Data":"a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955"} Nov 04 01:26:20.000216 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.000195 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf" event={"ID":"82a77e5e-42bb-4b2e-a233-942284a5e1e9","Type":"ContainerDied","Data":"766a53b2e5b3a467b2ee66739d10414a3fc7a29fdf0549b0fa6bb17124f9f1eb"} Nov 04 01:26:20.001113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.001083 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:26:20.001526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.001509 4312 generic.go:356] "Generic (PLEG): container finished" podID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerID="2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b" exitCode=0 Nov 04 01:26:20.001621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.001608 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" event={"ID":"51f907ae-3f89-40f0-b5be-96fba4eff5c1","Type":"ContainerDied","Data":"2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b"} Nov 04 01:26:20.001669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.001640 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" event={"ID":"51f907ae-3f89-40f0-b5be-96fba4eff5c1","Type":"ContainerDied","Data":"c38f2ca7874aee052bf82e3388364ecc6c2e9fe52504d920cef896d42cbb92c1"} Nov 04 01:26:20.001709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.001698 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk" Nov 04 01:26:20.025695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.025624 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-kube-api-access-r7pb9\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.025695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.025688 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istiod-ca-cert\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.025886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.025729 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638bc5b7-a66e-43d9-8053-b754b0b00afc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638bc5b7-a66e-43d9-8053-b754b0b00afc-istiod-ca-cert\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.025886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.025818 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-token\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-token\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.025886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.025856 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-certs\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.026059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.025929 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-data\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.025962 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-certs\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026065 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-token\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-token\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.026153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026102 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-credential-socket\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.026250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026193 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-podinfo\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.026250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026239 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b5c39f-d773-4ade-843d-2eeeee33361a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b5c39f-d773-4ade-843d-2eeeee33361a-istiod-ca-cert\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026308 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-podinfo\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026370 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-token\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-token\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026433 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-credential-socket\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026465 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-podinfo\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.026586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026497 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-kube-api-access-p9t97\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-kube-api-access-p9t97\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026538 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-socket\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.026586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026577 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-socket\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.026750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026608 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-socket\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.026750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026661 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-data\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.026750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026721 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-kube-api-access-hrrvx\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-kube-api-access-hrrvx\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.026894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026773 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-data\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.026894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026815 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-credential-socket\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.026894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026865 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-envoy\") pod \"36b5c39f-d773-4ade-843d-2eeeee33361a\" (UID: \"36b5c39f-d773-4ade-843d-2eeeee33361a\") " Nov 04 01:26:20.027043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.026971 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-envoy\") pod \"638bc5b7-a66e-43d9-8053-b754b0b00afc\" (UID: \"638bc5b7-a66e-43d9-8053-b754b0b00afc\") " Nov 04 01:26:20.027043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.027010 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-envoy\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.027150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.027086 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-certs\") pod \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\" (UID: \"82a77e5e-42bb-4b2e-a233-942284a5e1e9\") " Nov 04 01:26:20.028218 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.027785 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-credential-socket" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.028218 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.028054 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-certs" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.028783 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.028913 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5168ab-db6a-423c-81e8-80011c9d13f9-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.028934 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.028951 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2kc7\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-kube-api-access-m2kc7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.029081 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.029102 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.029120 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.029277 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8860371f-a537-41c7-ae99-7d5c165baed9-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.029360 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.032635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.029379 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8860371f-a537-41c7-ae99-7d5c165baed9-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.034802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.034133 4312 generic.go:356] "Generic (PLEG): container finished" podID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerID="e524f96396a9fe01c9002bfd5d5cfd648009ffe3fcddd900ba674ab7beac4686" exitCode=0 Nov 04 01:26:20.034802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.034229 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" event={"ID":"71044d9c-6298-48db-b42b-6aab36d15d4d","Type":"ContainerDied","Data":"e524f96396a9fe01c9002bfd5d5cfd648009ffe3fcddd900ba674ab7beac4686"} Nov 04 01:26:20.035713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.035672 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-certs" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.035842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.035724 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hzql\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-kube-api-access-9hzql\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.035842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.035825 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481d9419-e346-444c-b319-a38160b62de7-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.035967 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pmw4\" (UniqueName: \"kubernetes.io/projected/8860371f-a537-41c7-ae99-7d5c165baed9-kube-api-access-5pmw4\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.035992 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036008 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036026 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5168ab-db6a-423c-81e8-80011c9d13f9-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036041 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036056 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9j9z\" (UniqueName: \"kubernetes.io/projected/481d9419-e346-444c-b319-a38160b62de7-kube-api-access-w9j9z\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036073 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036269 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481d9419-e346-444c-b319-a38160b62de7-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036288 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036320 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036350 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481d9419-e346-444c-b319-a38160b62de7-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036366 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036383 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5168ab-db6a-423c-81e8-80011c9d13f9-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036398 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8860371f-a537-41c7-ae99-7d5c165baed9-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.036731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036413 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f907ae-3f89-40f0-b5be-96fba4eff5c1-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.037039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.036873 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-certs" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.037094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.037062 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-data" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.037618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.037367 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b5c39f-d773-4ade-843d-2eeeee33361a-istiod-ca-cert" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.037948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.037713 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-credential-socket" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.040354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.039926 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-token" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.040354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.039982 4312 generic.go:356] "Generic (PLEG): container finished" podID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerID="8cd64a5f27260f300ff6a4454101d8a920120d7429ea5df4766d6bca392d3bcb" exitCode=0 Nov 04 01:26:20.040354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.040084 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" event={"ID":"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857","Type":"ContainerDied","Data":"8cd64a5f27260f300ff6a4454101d8a920120d7429ea5df4766d6bca392d3bcb"} Nov 04 01:26:20.040354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.040197 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.041886 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-podinfo" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.042181 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istiod-ca-cert" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.042520 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638bc5b7-a66e-43d9-8053-b754b0b00afc-istiod-ca-cert" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.042919 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-socket" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.043131 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-socket" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.043376 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-socket" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.043576 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-data" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.044432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.044181 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-data" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.046496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.046469 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-credential-socket" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.046603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.046576 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-podinfo" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.048198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.047593 4312 generic.go:356] "Generic (PLEG): container finished" podID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerID="b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128" exitCode=0 Nov 04 01:26:20.048198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.047748 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" event={"ID":"638bc5b7-a66e-43d9-8053-b754b0b00afc","Type":"ContainerDied","Data":"b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128"} Nov 04 01:26:20.048198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.047796 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" event={"ID":"638bc5b7-a66e-43d9-8053-b754b0b00afc","Type":"ContainerDied","Data":"d5e32d8cfc1b7f7ae0d77e6cf3a7bce449436ee2707733b6abcbbb41c788d73f"} Nov 04 01:26:20.048198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.047905 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv" Nov 04 01:26:20.048993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.048687 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-kube-api-access-p9t97" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "kube-api-access-p9t97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.053520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.052658 4312 generic.go:356] "Generic (PLEG): container finished" podID="481d9419-e346-444c-b319-a38160b62de7" containerID="543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6" exitCode=0 Nov 04 01:26:20.053520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.052755 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" event={"ID":"481d9419-e346-444c-b319-a38160b62de7","Type":"ContainerDied","Data":"543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6"} Nov 04 01:26:20.053520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.052802 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" event={"ID":"481d9419-e346-444c-b319-a38160b62de7","Type":"ContainerDied","Data":"316adb800f62452a5bc32bc9799d39b2b3a13936c0927cda179e8486134eaecf"} Nov 04 01:26:20.053520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.052902 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:26:20.056588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.053698 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-token" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.056588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.053743 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-kube-api-access-hrrvx" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "kube-api-access-hrrvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.056588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.053848 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-podinfo" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.056588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.053978 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-kube-api-access-r7pb9" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "kube-api-access-r7pb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.056904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.056882 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-envoy" pod "82a77e5e-42bb-4b2e-a233-942284a5e1e9" (UID: "82a77e5e-42bb-4b2e-a233-942284a5e1e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.057005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.056946 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-envoy" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.057102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.056953 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-envoy" pod "638bc5b7-a66e-43d9-8053-b754b0b00afc" (UID: "638bc5b7-a66e-43d9-8053-b754b0b00afc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.057194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.057027 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-token" pod "36b5c39f-d773-4ade-843d-2eeeee33361a" (UID: "36b5c39f-d773-4ade-843d-2eeeee33361a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.060436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.058940 4312 generic.go:356] "Generic (PLEG): container finished" podID="8860371f-a537-41c7-ae99-7d5c165baed9" containerID="265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c" exitCode=0 Nov 04 01:26:20.060436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.059257 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:26:20.060436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.059413 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" event={"ID":"8860371f-a537-41c7-ae99-7d5c165baed9","Type":"ContainerDied","Data":"265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c"} Nov 04 01:26:20.060436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.059548 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" event={"ID":"8860371f-a537-41c7-ae99-7d5c165baed9","Type":"ContainerDied","Data":"01ad24c2b407ed3b09af4794c402ef17e4748c87ee3ea4a702d7eb983261738d"} Nov 04 01:26:20.065955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.065929 4312 generic.go:356] "Generic (PLEG): container finished" podID="2550f115-5ac3-4059-a467-83b06601cda5" containerID="3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0" exitCode=0 Nov 04 01:26:20.066061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.065979 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" Nov 04 01:26:20.066061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.066009 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" event={"ID":"2550f115-5ac3-4059-a467-83b06601cda5","Type":"ContainerDied","Data":"3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0"} Nov 04 01:26:20.066149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.066070 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929" event={"ID":"2550f115-5ac3-4059-a467-83b06601cda5","Type":"ContainerDied","Data":"6a64b7bddbedabc8b73ccfb1e6005ac052e24d3ca6ae97894c1d80a2faca7b70"} Nov 04 01:26:20.068059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.068037 4312 generic.go:356] "Generic (PLEG): container finished" podID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerID="2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623" exitCode=0 Nov 04 01:26:20.068257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.068117 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" event={"ID":"36b5c39f-d773-4ade-843d-2eeeee33361a","Type":"ContainerDied","Data":"2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623"} Nov 04 01:26:20.068487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.068289 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" event={"ID":"36b5c39f-d773-4ade-843d-2eeeee33361a","Type":"ContainerDied","Data":"d6c6e72d067a2a6334270b65e29f092fbdb686444d5d7d23ec0bfa833dea497a"} Nov 04 01:26:20.068487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.068110 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984" Nov 04 01:26:20.092695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.092527 4312 generic.go:356] "Generic (PLEG): container finished" podID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerID="6771ed881ea0585229b0e34a110ac47302f6233bbc1ebbde58f83926a45eb197" exitCode=0 Nov 04 01:26:20.093045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.092926 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" event={"ID":"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55","Type":"ContainerDied","Data":"6771ed881ea0585229b0e34a110ac47302f6233bbc1ebbde58f83926a45eb197"} Nov 04 01:26:20.143846 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.143806 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-data\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.143987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.143950 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-certs\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144055 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-istio-token\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-istio-token\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.144136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144092 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-credential-socket\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144154 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2550f115-5ac3-4059-a467-83b06601cda5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2550f115-5ac3-4059-a467-83b06601cda5-istiod-ca-cert\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.144255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144202 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istiod-ca-cert\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144249 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-envoy\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144286 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-kube-api-access-f9t5x\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-kube-api-access-f9t5x\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.144442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144374 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-kube-api-access-mfwmd\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-kube-api-access-mfwmd\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144404 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-envoy\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.144561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144438 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-data\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144493 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-token\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-token\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144559 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2550f115-5ac3-4059-a467-83b06601cda5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2550f115-5ac3-4059-a467-83b06601cda5-istio-podinfo\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.144668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144618 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-credential-socket\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.144781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144748 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-socket\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.144840 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144809 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-podinfo\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144838 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-socket\") pod \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\" (UID: \"869fa5f8-76a7-4a0d-bb9f-5c1c83abf857\") " Nov 04 01:26:20.144895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144862 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istiod-ca-cert" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.145011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.144942 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-certs\") pod \"2550f115-5ac3-4059-a467-83b06601cda5\" (UID: \"2550f115-5ac3-4059-a467-83b06601cda5\") " Nov 04 01:26:20.145657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.145624 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-socket" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148110 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7pb9\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-kube-api-access-r7pb9\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148147 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148163 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638bc5b7-a66e-43d9-8053-b754b0b00afc-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148180 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148194 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148207 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148223 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148235 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148244 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148253 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148263 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148287 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b5c39f-d773-4ade-843d-2eeeee33361a-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148299 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148308 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148317 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148326 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9t97\" (UniqueName: \"kubernetes.io/projected/36b5c39f-d773-4ade-843d-2eeeee33361a-kube-api-access-p9t97\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148352 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148365 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148378 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148388 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148399 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrvx\" (UniqueName: \"kubernetes.io/projected/82a77e5e-42bb-4b2e-a233-942284a5e1e9-kube-api-access-hrrvx\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148408 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148416 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148425 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148434 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b5c39f-d773-4ade-843d-2eeeee33361a-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148445 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638bc5b7-a66e-43d9-8053-b754b0b00afc-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.148611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.148461 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a77e5e-42bb-4b2e-a233-942284a5e1e9-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.150374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.149099 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2550f115-5ac3-4059-a467-83b06601cda5-istiod-ca-cert" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.150374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.149556 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-certs" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.151230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.150554 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-credential-socket" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.151682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.151390 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-credential-socket" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.155464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.154583 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-data" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.155464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.155030 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-certs" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.155464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.155155 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-data" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.155690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.155552 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-socket" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.156752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.156511 4312 scope.go:122] "RemoveContainer" containerID="ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0" Nov 04 01:26:20.158643 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.158618 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-podinfo" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.160488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.160460 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-envoy" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.161028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.160996 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-kube-api-access-mfwmd" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "kube-api-access-mfwmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.161797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.161777 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-istio-token" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.165236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.164950 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-envoy" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.165991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.165963 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-token" pod "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" (UID: "869fa5f8-76a7-4a0d-bb9f-5c1c83abf857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.169819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.169518 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2550f115-5ac3-4059-a467-83b06601cda5-istio-podinfo" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.176995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.176950 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-kube-api-access-f9t5x" pod "2550f115-5ac3-4059-a467-83b06601cda5" (UID: "2550f115-5ac3-4059-a467-83b06601cda5"). InnerVolumeSpecName "kube-api-access-f9t5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.222351 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.222304 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:26:20.225583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.225557 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:26:20.226264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.226239 4312 scope.go:122] "RemoveContainer" containerID="7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf" Nov 04 01:26:20.226613 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.226585 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf\": container with ID starting with 7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf not found: ID does not exist" containerID="7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf" Nov 04 01:26:20.226699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.226627 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf"} err="failed to get container status \"7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf\": rpc error: code = NotFound desc = could not find container \"7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf\": container with ID starting with 7e4c72798f3ebdace275d0d92f398fc1fa9323eb72abdd8f34b3d44eabf174cf not found: ID does not exist" Nov 04 01:26:20.226699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.226655 4312 scope.go:122] "RemoveContainer" containerID="ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0" Nov 04 01:26:20.226932 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.226908 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0\": container with ID starting with ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0 not found: ID does not exist" containerID="ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0" Nov 04 01:26:20.227013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.226940 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0"} err="failed to get container status \"ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0\": rpc error: code = NotFound desc = could not find container \"ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0\": container with ID starting with ea7ca9f14c1510122e87694e5be5868c4cca616c382c9807f746a8290426abd0 not found: ID does not exist" Nov 04 01:26:20.227013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.226956 4312 scope.go:122] "RemoveContainer" containerID="a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955" Nov 04 01:26:20.234968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.234856 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:26:20.252034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.252005 4312 scope.go:122] "RemoveContainer" containerID="a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955" Nov 04 01:26:20.253088 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.253061 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955\": container with ID starting with a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955 not found: ID does not exist" containerID="a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955" Nov 04 01:26:20.253148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.253125 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955"} err="failed to get container status \"a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955\": rpc error: code = NotFound desc = could not find container \"a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955\": container with ID starting with a284b97156ccc54f78298bcc8c7c89a2c40904dd3acb5af50bc2e2e76fb49955 not found: ID does not exist" Nov 04 01:26:20.253215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.253153 4312 scope.go:122] "RemoveContainer" containerID="2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b" Nov 04 01:26:20.254465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254438 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-kube-api-access-p9bpm\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-kube-api-access-p9bpm\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.254578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254487 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-certs\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.254578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254534 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-kube-api-access-jhqpj\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-kube-api-access-jhqpj\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.254578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254571 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-socket\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.254810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254605 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-token\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-token\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.254810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254642 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-socket\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.254810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254678 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-envoy\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.254810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254788 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-token\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-token\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.255067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254823 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-istio-token\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-istio-token\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.255067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254907 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-data\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.255067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.254955 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33dee613-4f76-4d13-9700-22a0d178c024-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33dee613-4f76-4d13-9700-22a0d178c024-istio-podinfo\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.255067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255011 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33dee613-4f76-4d13-9700-22a0d178c024-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33dee613-4f76-4d13-9700-22a0d178c024-istiod-ca-cert\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.255067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255060 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-credential-socket\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.255394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255106 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-envoy\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.255394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255160 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-data\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.255394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255197 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-credential-socket\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.255394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255291 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-certs\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.255394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255346 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-credential-socket\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.255394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255389 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-socket\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.255789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.255454 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-kube-api-access-9sx4q\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-kube-api-access-9sx4q\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.256418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256395 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwmd\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-kube-api-access-mfwmd\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.256555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256539 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.256650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256637 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.256759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256746 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.256847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256836 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2550f115-5ac3-4059-a467-83b06601cda5-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.256930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256918 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256999 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257084 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257173 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257261 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2550f115-5ac3-4059-a467-83b06601cda5-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257372 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257459 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257567 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257657 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2550f115-5ac3-4059-a467-83b06601cda5-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257754 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.257838 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9t5x\" (UniqueName: \"kubernetes.io/projected/2550f115-5ac3-4059-a467-83b06601cda5-kube-api-access-f9t5x\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.257924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256551 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-data" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.258003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.256764 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-socket" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.258529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.258506 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-credential-socket" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.259191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.259168 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dee613-4f76-4d13-9700-22a0d178c024-istiod-ca-cert" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.259400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.259375 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-envoy" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.259944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.259616 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-credential-socket" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.259944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.259786 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-certs" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.260114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.259972 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-socket" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.260180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.260144 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-certs" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.260463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.260440 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-credential-socket" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.266377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.266353 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-socket" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.266706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.266686 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-data" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.266828 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.266737 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-istio-token" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.269049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.269013 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-kube-api-access-9sx4q" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "kube-api-access-9sx4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.271483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.271459 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-kube-api-access-p9bpm" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "kube-api-access-p9bpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.271601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.271496 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-kube-api-access-jhqpj" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "kube-api-access-jhqpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.271854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.271694 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33dee613-4f76-4d13-9700-22a0d178c024-istio-podinfo" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.271854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.271743 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-token" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.273627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.273592 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-envoy" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.273755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.273653 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-token" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:20.292155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.292128 4312 scope.go:122] "RemoveContainer" containerID="2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b" Nov 04 01:26:20.297562 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.297516 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b\": container with ID starting with 2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b not found: ID does not exist" containerID="2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b" Nov 04 01:26:20.297715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.297572 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b"} err="failed to get container status \"2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b\": rpc error: code = NotFound desc = could not find container \"2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b\": container with ID starting with 2b3ce7c1a754f606fa198bae14d8ba1335a114b44ce1b72656d353d2b101c83b not found: ID does not exist" Nov 04 01:26:20.297715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.297615 4312 scope.go:122] "RemoveContainer" containerID="8cd64a5f27260f300ff6a4454101d8a920120d7429ea5df4766d6bca392d3bcb" Nov 04 01:26:20.304747 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.304717 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984"] Nov 04 01:26:20.316373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.316318 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-27984"] Nov 04 01:26:20.320714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.320690 4312 scope.go:122] "RemoveContainer" containerID="b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128" Nov 04 01:26:20.327275 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.327229 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5"] Nov 04 01:26:20.333682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.333659 4312 scope.go:122] "RemoveContainer" containerID="4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0" Nov 04 01:26:20.335434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.335412 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-hxmb5"] Nov 04 01:26:20.354855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.354736 4312 scope.go:122] "RemoveContainer" containerID="b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128" Nov 04 01:26:20.359960 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.359933 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71044d9c-6298-48db-b42b-6aab36d15d4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71044d9c-6298-48db-b42b-6aab36d15d4d-istiod-ca-cert\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.360180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.360164 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-envoy\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.362374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.360430 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71044d9c-6298-48db-b42b-6aab36d15d4d-istiod-ca-cert" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.362536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.362519 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istiod-ca-cert\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.362635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.362557 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-certs\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.362635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.362614 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-podinfo\") pod \"71044d9c-6298-48db-b42b-6aab36d15d4d\" (UID: \"71044d9c-6298-48db-b42b-6aab36d15d4d\") " Nov 04 01:26:20.362746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.362648 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-data\") pod \"33dee613-4f76-4d13-9700-22a0d178c024\" (UID: \"33dee613-4f76-4d13-9700-22a0d178c024\") " Nov 04 01:26:20.362746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.362676 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-podinfo\") pod \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\" (UID: \"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55\") " Nov 04 01:26:20.363676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.362879 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-certs" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.363676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.363129 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istiod-ca-cert" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:20.363676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.363431 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-data" pod "33dee613-4f76-4d13-9700-22a0d178c024" (UID: "33dee613-4f76-4d13-9700-22a0d178c024"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.365033 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.364850 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128\": container with ID starting with b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128 not found: ID does not exist" containerID="b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128" Nov 04 01:26:20.365033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.364891 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128"} err="failed to get container status \"b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128\": rpc error: code = NotFound desc = could not find container \"b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128\": container with ID starting with b7d17879717bad21af624d75572794c832135802d5489381fccf4d7435808128 not found: ID does not exist" Nov 04 01:26:20.365033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.364927 4312 scope.go:122] "RemoveContainer" containerID="4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0" Nov 04 01:26:20.365724 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.365561 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0\": container with ID starting with 4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0 not found: ID does not exist" containerID="4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0" Nov 04 01:26:20.365724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.365598 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0"} err="failed to get container status \"4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0\": rpc error: code = NotFound desc = could not find container \"4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0\": container with ID starting with 4f195f38af942310d0c45619036aefa2578baa80a32940947e38092c071319a0 not found: ID does not exist" Nov 04 01:26:20.365724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.365618 4312 scope.go:122] "RemoveContainer" containerID="543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6" Nov 04 01:26:20.366661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366636 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366666 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33dee613-4f76-4d13-9700-22a0d178c024-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366681 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33dee613-4f76-4d13-9700-22a0d178c024-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366694 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366710 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366722 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366735 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366750 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366762 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366775 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366788 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sx4q\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-kube-api-access-9sx4q\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.366783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366801 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71044d9c-6298-48db-b42b-6aab36d15d4d-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366814 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.366828 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.367126 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-podinfo" pod "71044d9c-6298-48db-b42b-6aab36d15d4d" (UID: "71044d9c-6298-48db-b42b-6aab36d15d4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.368401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.367129 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.367216 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-envoy" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:20.368401 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368324 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9bpm\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-kube-api-access-p9bpm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368414 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368432 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqpj\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-kube-api-access-jhqpj\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368447 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33dee613-4f76-4d13-9700-22a0d178c024-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368462 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368475 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368490 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368503 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368516 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33dee613-4f76-4d13-9700-22a0d178c024-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.368859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.368667 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-podinfo" pod "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" (UID: "c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:20.385072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.382149 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk"] Nov 04 01:26:20.390687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.390571 4312 scope.go:122] "RemoveContainer" containerID="543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6" Nov 04 01:26:20.391167 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.391137 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6\": container with ID starting with 543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6 not found: ID does not exist" containerID="543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6" Nov 04 01:26:20.391328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.391296 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6"} err="failed to get container status \"543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6\": rpc error: code = NotFound desc = could not find container \"543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6\": container with ID starting with 543b090bbbc3c7609bd54bc3656d5fa2bdc52b9dd192c2f7268e7a1e3f08c1b6 not found: ID does not exist" Nov 04 01:26:20.391328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.391329 4312 scope.go:122] "RemoveContainer" containerID="265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c" Nov 04 01:26:20.399644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.399616 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-kmqhk"] Nov 04 01:26:20.403030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.403013 4312 scope.go:122] "RemoveContainer" containerID="265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c" Nov 04 01:26:20.403427 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.403390 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c\": container with ID starting with 265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c not found: ID does not exist" containerID="265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c" Nov 04 01:26:20.403527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.403428 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c"} err="failed to get container status \"265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c\": rpc error: code = NotFound desc = could not find container \"265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c\": container with ID starting with 265426c33a1bb397daa10af52a84adf099d20b8ce3493f9f53aea8483e9a507c not found: ID does not exist" Nov 04 01:26:20.403527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.403454 4312 scope.go:122] "RemoveContainer" containerID="3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0" Nov 04 01:26:20.418371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.415721 4312 scope.go:122] "RemoveContainer" containerID="3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0" Nov 04 01:26:20.418371 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.416367 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0\": container with ID starting with 3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0 not found: ID does not exist" containerID="3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0" Nov 04 01:26:20.418371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.416402 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0"} err="failed to get container status \"3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0\": rpc error: code = NotFound desc = could not find container \"3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0\": container with ID starting with 3094b2554bfe189273a41325a350d501acab9a8a620ba6f6c8e182cf69ec53d0 not found: ID does not exist" Nov 04 01:26:20.418371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.416423 4312 scope.go:122] "RemoveContainer" containerID="2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623" Nov 04 01:26:20.418371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.418264 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv"] Nov 04 01:26:20.424317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.424295 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-69fnv"] Nov 04 01:26:20.426945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.426929 4312 scope.go:122] "RemoveContainer" containerID="2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623" Nov 04 01:26:20.427214 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:20.427196 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623\": container with ID starting with 2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623 not found: ID does not exist" containerID="2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623" Nov 04 01:26:20.427274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.427225 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623"} err="failed to get container status \"2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623\": rpc error: code = NotFound desc = could not find container \"2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623\": container with ID starting with 2ea5aba3f33683272e0e23d3299befd01e16c68e0371180092622d6ad9bbc623 not found: ID does not exist" Nov 04 01:26:20.433622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.433600 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf"] Nov 04 01:26:20.440094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.440060 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-w8nhf"] Nov 04 01:26:20.452702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.452678 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb"] Nov 04 01:26:20.461994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.461945 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-dn5gb"] Nov 04 01:26:20.474879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.474834 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.475107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.475088 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71044d9c-6298-48db-b42b-6aab36d15d4d-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.475226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.475193 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.484690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.484648 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929"] Nov 04 01:26:20.489550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:20.489522 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-sx929"] Nov 04 01:26:21.106462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.106435 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" Nov 04 01:26:21.106915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.106432 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5" event={"ID":"33dee613-4f76-4d13-9700-22a0d178c024","Type":"ContainerDied","Data":"892b8ed7d475fdc98f68354c7eaa0febf3b24c7fd69b07d9095ecad92ab234b1"} Nov 04 01:26:21.106915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.106594 4312 scope.go:122] "RemoveContainer" containerID="9a8b7c1d965b0a15c1f29442e05227586381cecafa1f2aec9d492371b3fcf13f" Nov 04 01:26:21.112091 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.112071 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" Nov 04 01:26:21.112225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.112073 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r" event={"ID":"71044d9c-6298-48db-b42b-6aab36d15d4d","Type":"ContainerDied","Data":"370b0e1a2b3b6e7db12aa0e0f09bd4f5dac3624d025ae3678f005a48f9218520"} Nov 04 01:26:21.120680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.120660 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" Nov 04 01:26:21.120783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.120666 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9" event={"ID":"c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55","Type":"ContainerDied","Data":"5df9d6eca478d38dcea2cea8fdd680f365f1a487dc7531cf934c62abd1b99498"} Nov 04 01:26:21.126492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.126465 4312 scope.go:122] "RemoveContainer" containerID="e524f96396a9fe01c9002bfd5d5cfd648009ffe3fcddd900ba674ab7beac4686" Nov 04 01:26:21.137543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.137498 4312 scope.go:122] "RemoveContainer" containerID="6771ed881ea0585229b0e34a110ac47302f6233bbc1ebbde58f83926a45eb197" Nov 04 01:26:21.142501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.142476 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5"] Nov 04 01:26:21.152390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.152365 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-mqxg5"] Nov 04 01:26:21.165247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.165220 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r"] Nov 04 01:26:21.171524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.171498 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-7tv6r"] Nov 04 01:26:21.183784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.183758 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9"] Nov 04 01:26:21.191077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:21.191047 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-n6bx9"] Nov 04 01:26:22.110418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.110384 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2550f115-5ac3-4059-a467-83b06601cda5" path="/var/lib/kubelet/pods/2550f115-5ac3-4059-a467-83b06601cda5/volumes" Nov 04 01:26:22.111117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.111095 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33dee613-4f76-4d13-9700-22a0d178c024" path="/var/lib/kubelet/pods/33dee613-4f76-4d13-9700-22a0d178c024/volumes" Nov 04 01:26:22.111666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.111644 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" path="/var/lib/kubelet/pods/36b5c39f-d773-4ade-843d-2eeeee33361a/volumes" Nov 04 01:26:22.112495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.112476 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" path="/var/lib/kubelet/pods/51f907ae-3f89-40f0-b5be-96fba4eff5c1/volumes" Nov 04 01:26:22.112993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.112974 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" path="/var/lib/kubelet/pods/638bc5b7-a66e-43d9-8053-b754b0b00afc/volumes" Nov 04 01:26:22.113525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.113504 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" path="/var/lib/kubelet/pods/71044d9c-6298-48db-b42b-6aab36d15d4d/volumes" Nov 04 01:26:22.114250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.114234 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" path="/var/lib/kubelet/pods/82a77e5e-42bb-4b2e-a233-942284a5e1e9/volumes" Nov 04 01:26:22.114695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.114681 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" path="/var/lib/kubelet/pods/869fa5f8-76a7-4a0d-bb9f-5c1c83abf857/volumes" Nov 04 01:26:22.115472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.115456 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" path="/var/lib/kubelet/pods/c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55/volumes" Nov 04 01:26:22.115954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.115930 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" path="/var/lib/kubelet/pods/dc5168ab-db6a-423c-81e8-80011c9d13f9/volumes" Nov 04 01:26:22.504591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.504567 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:26:22.612743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.612705 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-envoy\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.612910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.612838 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6250006e-8963-4565-91e4-6d09ceadf762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6250006e-8963-4565-91e4-6d09ceadf762-istiod-ca-cert\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613004 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-kube-api-access-fjrkv\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-kube-api-access-fjrkv\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613045 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-certs\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613120 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-data\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613193 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6250006e-8963-4565-91e4-6d09ceadf762-istiod-ca-cert" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:22.613278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613214 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-istio-token\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-istio-token\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613387 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-certs" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:22.613552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613453 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6250006e-8963-4565-91e4-6d09ceadf762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6250006e-8963-4565-91e4-6d09ceadf762-istio-podinfo\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613483 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-data" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:22.613552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613512 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-socket\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613557 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-credential-socket\") pod \"6250006e-8963-4565-91e4-6d09ceadf762\" (UID: \"6250006e-8963-4565-91e4-6d09ceadf762\") " Nov 04 01:26:22.613784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613717 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-socket" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:22.613868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.613838 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-credential-socket" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:22.614449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.614425 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6250006e-8963-4565-91e4-6d09ceadf762-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.614503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.614456 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.614503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.614473 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.614503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.614486 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.614598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.614500 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.614929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.614911 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-envoy" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:22.615545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.615529 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-istio-token" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:22.615675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.615652 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-kube-api-access-fjrkv" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "kube-api-access-fjrkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:22.615735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.615701 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6250006e-8963-4565-91e4-6d09ceadf762-istio-podinfo" pod "6250006e-8963-4565-91e4-6d09ceadf762" (UID: "6250006e-8963-4565-91e4-6d09ceadf762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:22.716402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.716282 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjrkv\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-kube-api-access-fjrkv\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.716402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.716317 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6250006e-8963-4565-91e4-6d09ceadf762-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.716402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.716346 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6250006e-8963-4565-91e4-6d09ceadf762-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:22.716402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:22.716359 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6250006e-8963-4565-91e4-6d09ceadf762-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:23.150609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.150573 4312 generic.go:356] "Generic (PLEG): container finished" podID="6250006e-8963-4565-91e4-6d09ceadf762" containerID="58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0" exitCode=0 Nov 04 01:26:23.150982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.150635 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" Nov 04 01:26:23.150982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.150653 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" event={"ID":"6250006e-8963-4565-91e4-6d09ceadf762","Type":"ContainerDied","Data":"58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0"} Nov 04 01:26:23.150982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.150736 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk" event={"ID":"6250006e-8963-4565-91e4-6d09ceadf762","Type":"ContainerDied","Data":"86e73587fa99f44c836462c7333c7d9ac7cf331571b4efb1c65398bb5acbc66d"} Nov 04 01:26:23.150982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.150761 4312 scope.go:122] "RemoveContainer" containerID="58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0" Nov 04 01:26:23.162904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.162885 4312 scope.go:122] "RemoveContainer" containerID="58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0" Nov 04 01:26:23.163197 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:23.163173 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0\": container with ID starting with 58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0 not found: ID does not exist" containerID="58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0" Nov 04 01:26:23.163248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.163206 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0"} err="failed to get container status \"58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0\": rpc error: code = NotFound desc = could not find container \"58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0\": container with ID starting with 58ff5766ffce4ae562c3756fff01d5e42bcef6da0c10c0e8849c72cd2dc6b0f0 not found: ID does not exist" Nov 04 01:26:23.177457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.177431 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk"] Nov 04 01:26:23.181440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:23.181420 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-943d4a-openshift-default-d6f47645c-rr6nk"] Nov 04 01:26:24.110210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:24.110177 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6250006e-8963-4565-91e4-6d09ceadf762" path="/var/lib/kubelet/pods/6250006e-8963-4565-91e4-6d09ceadf762/volumes" Nov 04 01:26:24.474817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:24.474740 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g"] Nov 04 01:26:29.459472 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:29.459433 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:26:33.672709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:33.672674 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km"] Nov 04 01:26:33.684870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:33.684843 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv"] Nov 04 01:26:38.673685 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:38.673640 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" podUID="2bc27530-aef3-4e8d-83b7-ebc205aa50ec" containerName="payload-processing" containerID="cri-o://b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9" gracePeriod=25 Nov 04 01:26:38.686109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:38.686071 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" podUID="2e617c76-3e92-4f95-81b0-e0a9977f38a1" containerName="payload-pre-processing" containerID="cri-o://0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930" gracePeriod=25 Nov 04 01:26:39.001720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.001696 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:26:39.052149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.052121 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:26:39.110959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.110922 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-kube-api-access-l795s\" (UniqueName: \"kubernetes.io/projected/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-kube-api-access-l795s\") pod \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " Nov 04 01:26:39.111211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.111188 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-plugins-config-volume\") pod \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\" (UID: \"2bc27530-aef3-4e8d-83b7-ebc205aa50ec\") " Nov 04 01:26:39.111920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.111892 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-plugins-config-volume" pod "2bc27530-aef3-4e8d-83b7-ebc205aa50ec" (UID: "2bc27530-aef3-4e8d-83b7-ebc205aa50ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:39.112267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.112240 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.114140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.114105 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-kube-api-access-l795s" pod "2bc27530-aef3-4e8d-83b7-ebc205aa50ec" (UID: "2bc27530-aef3-4e8d-83b7-ebc205aa50ec"). InnerVolumeSpecName "kube-api-access-l795s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:39.213295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.213214 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e617c76-3e92-4f95-81b0-e0a9977f38a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e617c76-3e92-4f95-81b0-e0a9977f38a1-plugins-config-volume\") pod \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " Nov 04 01:26:39.213295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.213278 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e617c76-3e92-4f95-81b0-e0a9977f38a1-kube-api-access-t642l\" (UniqueName: \"kubernetes.io/projected/2e617c76-3e92-4f95-81b0-e0a9977f38a1-kube-api-access-t642l\") pod \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\" (UID: \"2e617c76-3e92-4f95-81b0-e0a9977f38a1\") " Nov 04 01:26:39.213685 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.213657 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e617c76-3e92-4f95-81b0-e0a9977f38a1-plugins-config-volume" pod "2e617c76-3e92-4f95-81b0-e0a9977f38a1" (UID: "2e617c76-3e92-4f95-81b0-e0a9977f38a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:39.213984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.213965 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l795s\" (UniqueName: \"kubernetes.io/projected/2bc27530-aef3-4e8d-83b7-ebc205aa50ec-kube-api-access-l795s\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.214063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.213989 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e617c76-3e92-4f95-81b0-e0a9977f38a1-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.215358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.215319 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e617c76-3e92-4f95-81b0-e0a9977f38a1-kube-api-access-t642l" pod "2e617c76-3e92-4f95-81b0-e0a9977f38a1" (UID: "2e617c76-3e92-4f95-81b0-e0a9977f38a1"). InnerVolumeSpecName "kube-api-access-t642l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:39.300945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.300907 4312 generic.go:356] "Generic (PLEG): container finished" podID="2bc27530-aef3-4e8d-83b7-ebc205aa50ec" containerID="b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9" exitCode=0 Nov 04 01:26:39.301129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.300965 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" Nov 04 01:26:39.301225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.301198 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" event={"ID":"2bc27530-aef3-4e8d-83b7-ebc205aa50ec","Type":"ContainerDied","Data":"b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9"} Nov 04 01:26:39.301300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.301261 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km" event={"ID":"2bc27530-aef3-4e8d-83b7-ebc205aa50ec","Type":"ContainerDied","Data":"53a687d81fea325e1ebea5a773e7de5f86f433f014d43c2bc6a94f10de1c8367"} Nov 04 01:26:39.301300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.301292 4312 scope.go:122] "RemoveContainer" containerID="b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9" Nov 04 01:26:39.305492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.303733 4312 generic.go:356] "Generic (PLEG): container finished" podID="2e617c76-3e92-4f95-81b0-e0a9977f38a1" containerID="0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930" exitCode=0 Nov 04 01:26:39.305492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.303800 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" event={"ID":"2e617c76-3e92-4f95-81b0-e0a9977f38a1","Type":"ContainerDied","Data":"0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930"} Nov 04 01:26:39.305492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.303842 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" event={"ID":"2e617c76-3e92-4f95-81b0-e0a9977f38a1","Type":"ContainerDied","Data":"fe5f9e8497f2443c751987a0f74c7a5701929c8e0117884f5bedccae6461e761"} Nov 04 01:26:39.305492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.303940 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv" Nov 04 01:26:39.315682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.315655 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t642l\" (UniqueName: \"kubernetes.io/projected/2e617c76-3e92-4f95-81b0-e0a9977f38a1-kube-api-access-t642l\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.316395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.316307 4312 scope.go:122] "RemoveContainer" containerID="b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9" Nov 04 01:26:39.316656 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:39.316633 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9\": container with ID starting with b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9 not found: ID does not exist" containerID="b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9" Nov 04 01:26:39.316767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.316664 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9"} err="failed to get container status \"b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9\": rpc error: code = NotFound desc = could not find container \"b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9\": container with ID starting with b7d1d36f4d1f78e6dfe8e78af8a5fece56de681cebedfb9976685b66f39e1cd9 not found: ID does not exist" Nov 04 01:26:39.316767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.316679 4312 scope.go:122] "RemoveContainer" containerID="0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930" Nov 04 01:26:39.334258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.334235 4312 scope.go:122] "RemoveContainer" containerID="0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930" Nov 04 01:26:39.335471 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:39.334844 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930\": container with ID starting with 0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930 not found: ID does not exist" containerID="0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930" Nov 04 01:26:39.335471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.334881 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930"} err="failed to get container status \"0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930\": rpc error: code = NotFound desc = could not find container \"0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930\": container with ID starting with 0a00e048e970c2decf837e7398e79aa6e5e4b388eb08645c45438c780f458930 not found: ID does not exist" Nov 04 01:26:39.336982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.336957 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km"] Nov 04 01:26:39.341505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.341482 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fdd516-86cfd78fcb-v99km"] Nov 04 01:26:39.350208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.350183 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv"] Nov 04 01:26:39.354318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.354296 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fdd516-5ddbf65c55-b58hv"] Nov 04 01:26:39.497241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.497136 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" containerID="cri-o://5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb" gracePeriod=45 Nov 04 01:26:39.796508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.796486 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:26:39.822644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.822614 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-tmp-dir\") pod \"2d2d0569-9744-4daf-aa0e-6b06c891a357\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " Nov 04 01:26:39.822836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.822820 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-model-cache\") pod \"2d2d0569-9744-4daf-aa0e-6b06c891a357\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " Nov 04 01:26:39.822896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.822871 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2d0569-9744-4daf-aa0e-6b06c891a357-kube-api-access-bp8nw\" (UniqueName: \"kubernetes.io/projected/2d2d0569-9744-4daf-aa0e-6b06c891a357-kube-api-access-bp8nw\") pod \"2d2d0569-9744-4daf-aa0e-6b06c891a357\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " Nov 04 01:26:39.822945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.822897 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-dshm\") pod \"2d2d0569-9744-4daf-aa0e-6b06c891a357\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " Nov 04 01:26:39.822992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.822944 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-home\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-home\") pod \"2d2d0569-9744-4daf-aa0e-6b06c891a357\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " Nov 04 01:26:39.823052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.823041 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2d0569-9744-4daf-aa0e-6b06c891a357-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d0569-9744-4daf-aa0e-6b06c891a357-tls-certs\") pod \"2d2d0569-9744-4daf-aa0e-6b06c891a357\" (UID: \"2d2d0569-9744-4daf-aa0e-6b06c891a357\") " Nov 04 01:26:39.823392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.823358 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-model-cache" pod "2d2d0569-9744-4daf-aa0e-6b06c891a357" (UID: "2d2d0569-9744-4daf-aa0e-6b06c891a357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.823612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.823582 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-home" pod "2d2d0569-9744-4daf-aa0e-6b06c891a357" (UID: "2d2d0569-9744-4daf-aa0e-6b06c891a357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.823792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.823764 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-tmp-dir" pod "2d2d0569-9744-4daf-aa0e-6b06c891a357" (UID: "2d2d0569-9744-4daf-aa0e-6b06c891a357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.823925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.823907 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.824019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.824007 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.825759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.825727 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-dshm" pod "2d2d0569-9744-4daf-aa0e-6b06c891a357" (UID: "2d2d0569-9744-4daf-aa0e-6b06c891a357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.825866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.825843 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2d0569-9744-4daf-aa0e-6b06c891a357-tls-certs" pod "2d2d0569-9744-4daf-aa0e-6b06c891a357" (UID: "2d2d0569-9744-4daf-aa0e-6b06c891a357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:39.825925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.825892 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2d0569-9744-4daf-aa0e-6b06c891a357-kube-api-access-bp8nw" pod "2d2d0569-9744-4daf-aa0e-6b06c891a357" (UID: "2d2d0569-9744-4daf-aa0e-6b06c891a357"). InnerVolumeSpecName "kube-api-access-bp8nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:39.925389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.925354 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8nw\" (UniqueName: \"kubernetes.io/projected/2d2d0569-9744-4daf-aa0e-6b06c891a357-kube-api-access-bp8nw\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.925389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.925389 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.925590 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.925406 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d0569-9744-4daf-aa0e-6b06c891a357-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.925590 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:39.925447 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2d0569-9744-4daf-aa0e-6b06c891a357-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:40.111489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.111458 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc27530-aef3-4e8d-83b7-ebc205aa50ec" path="/var/lib/kubelet/pods/2bc27530-aef3-4e8d-83b7-ebc205aa50ec/volumes" Nov 04 01:26:40.111934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.111918 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e617c76-3e92-4f95-81b0-e0a9977f38a1" path="/var/lib/kubelet/pods/2e617c76-3e92-4f95-81b0-e0a9977f38a1/volumes" Nov 04 01:26:40.314505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.314468 4312 generic.go:356] "Generic (PLEG): container finished" podID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerID="5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb" exitCode=0 Nov 04 01:26:40.314661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.314517 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" Nov 04 01:26:40.314661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.314546 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" event={"ID":"2d2d0569-9744-4daf-aa0e-6b06c891a357","Type":"ContainerDied","Data":"5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb"} Nov 04 01:26:40.314661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.314589 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g" event={"ID":"2d2d0569-9744-4daf-aa0e-6b06c891a357","Type":"ContainerDied","Data":"2f120f479d94a98dd8ba868d261caea41f146835ff2ca95dacda6acab504c592"} Nov 04 01:26:40.314661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.314606 4312 scope.go:122] "RemoveContainer" containerID="5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb" Nov 04 01:26:40.326772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.326751 4312 scope.go:122] "RemoveContainer" containerID="5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb" Nov 04 01:26:40.327051 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:40.327033 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb\": container with ID starting with 5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb not found: ID does not exist" containerID="5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb" Nov 04 01:26:40.327121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.327059 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb"} err="failed to get container status \"5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb\": rpc error: code = NotFound desc = could not find container \"5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb\": container with ID starting with 5237385efa21a67a927dc0f6dadf48cef2ac349f6bce712a00e153fd8b1b1ccb not found: ID does not exist" Nov 04 01:26:40.335451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.335428 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g"] Nov 04 01:26:40.339507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:40.339486 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-9902c3/distinct-2-w6-9902c3-kserve-56854bcd94-w782g"] Nov 04 01:26:42.113248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:42.113218 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" path="/var/lib/kubelet/pods/2d2d0569-9744-4daf-aa0e-6b06c891a357/volumes" Nov 04 01:26:49.157167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:49.151504 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74"] Nov 04 01:26:49.157167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:49.151814 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" containerID="cri-o://ad60383a04bf84ac66ac187fde586eab6409e4f4e49e877c28221f9539e17598" gracePeriod=30 Nov 04 01:26:49.160726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:49.160699 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk"] Nov 04 01:26:49.160949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:49.160921 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" containerID="cri-o://17fde16744070272d4ef9a459aa86054c9299ebf6e439a9882907f463c4b9f6d" gracePeriod=30 Nov 04 01:26:49.180430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:49.180399 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2"] Nov 04 01:26:49.180651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:49.180627 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" containerID="cri-o://a75542098305c2347d93f520d4a090c0102febe9015b5804640881e0b107a92f" gracePeriod=30 Nov 04 01:26:50.215534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.215473 4312 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod481d9419-e346-444c-b319-a38160b62de7"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod481d9419-e346-444c-b319-a38160b62de7] : Timed out while waiting for systemd to remove kubepods-burstable-pod481d9419_e346_444c_b319_a38160b62de7.slice" Nov 04 01:26:50.215939 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:50.215542 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod481d9419-e346-444c-b319-a38160b62de7] : unable to destroy cgroup paths for cgroup [kubepods burstable pod481d9419-e346-444c-b319-a38160b62de7] : Timed out while waiting for systemd to remove kubepods-burstable-pod481d9419_e346_444c_b319_a38160b62de7.slice" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" podUID="481d9419-e346-444c-b319-a38160b62de7" Nov 04 01:26:50.218627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.218596 4312 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod8860371f-a537-41c7-ae99-7d5c165baed9"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod8860371f-a537-41c7-ae99-7d5c165baed9] : Timed out while waiting for systemd to remove kubepods-burstable-pod8860371f_a537_41c7_ae99_7d5c165baed9.slice" Nov 04 01:26:50.218627 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:50.218626 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod8860371f-a537-41c7-ae99-7d5c165baed9] : unable to destroy cgroup paths for cgroup [kubepods burstable pod8860371f-a537-41c7-ae99-7d5c165baed9] : Timed out while waiting for systemd to remove kubepods-burstable-pod8860371f_a537_41c7_ae99_7d5c165baed9.slice" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" Nov 04 01:26:50.440890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.440862 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv" Nov 04 01:26:50.441056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.440978 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg" Nov 04 01:26:50.503430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.503352 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv"] Nov 04 01:26:50.506918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.506893 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-r7gmv"] Nov 04 01:26:50.523755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.523712 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg"] Nov 04 01:26:50.533680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.533638 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-8c4xg"] Nov 04 01:26:50.657229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.657188 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx"] Nov 04 01:26:50.657814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:50.657764 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" podUID="f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" containerName="maas-api" containerID="cri-o://664b7d9fa9107a68ad4312302e725caf5556217cdf587c6861df8ef08d84bbd1" gracePeriod=30 Nov 04 01:26:51.456549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.451775 4312 generic.go:356] "Generic (PLEG): container finished" podID="f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" containerID="664b7d9fa9107a68ad4312302e725caf5556217cdf587c6861df8ef08d84bbd1" exitCode=0 Nov 04 01:26:51.456549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.451867 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" event={"ID":"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5","Type":"ContainerDied","Data":"664b7d9fa9107a68ad4312302e725caf5556217cdf587c6861df8ef08d84bbd1"} Nov 04 01:26:51.456549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.451903 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" event={"ID":"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5","Type":"ContainerDied","Data":"befac66a18186cd459077a730b981e7cb50c2e8b2e515a5e1300f63f11e9c3fe"} Nov 04 01:26:51.456549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.451925 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="befac66a18186cd459077a730b981e7cb50c2e8b2e515a5e1300f63f11e9c3fe" Nov 04 01:26:51.474282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.474250 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:26:51.538664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.538629 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd"] Nov 04 01:26:51.539870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539845 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2bc27530-aef3-4e8d-83b7-ebc205aa50ec" containerName="payload-processing" Nov 04 01:26:51.539944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539873 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc27530-aef3-4e8d-83b7-ebc205aa50ec" containerName="payload-processing" Nov 04 01:26:51.539944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539897 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" Nov 04 01:26:51.539944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539907 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" Nov 04 01:26:51.539944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539920 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" Nov 04 01:26:51.539944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539929 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539947 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539954 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539969 4312 cpu_manager.go:414] "containerMap: removing container" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539977 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539986 4312 cpu_manager.go:414] "containerMap: removing container" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.539994 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540028 4312 cpu_manager.go:414] "containerMap: removing container" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540036 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540048 4312 cpu_manager.go:414] "containerMap: removing container" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540055 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540065 4312 cpu_manager.go:414] "containerMap: removing container" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540073 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540093 4312 cpu_manager.go:414] "containerMap: removing container" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540100 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" Nov 04 01:26:51.540117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540120 4312 cpu_manager.go:414] "containerMap: removing container" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540127 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540149 4312 cpu_manager.go:414] "containerMap: removing container" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540156 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540170 4312 cpu_manager.go:414] "containerMap: removing container" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540178 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540190 4312 cpu_manager.go:414] "containerMap: removing container" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540197 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540206 4312 cpu_manager.go:414] "containerMap: removing container" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540213 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540227 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2e617c76-3e92-4f95-81b0-e0a9977f38a1" containerName="payload-pre-processing" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540233 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e617c76-3e92-4f95-81b0-e0a9977f38a1" containerName="payload-pre-processing" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540249 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540256 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540279 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" containerName="maas-api" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540286 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" containerName="maas-api" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540482 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="c098cc9b-21ac-4ece-a7a7-1ec5d6bf2c55" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540503 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540515 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a77e5e-42bb-4b2e-a233-942284a5e1e9" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540525 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b5c39f-d773-4ade-843d-2eeeee33361a" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540532 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="869fa5f8-76a7-4a0d-bb9f-5c1c83abf857" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540545 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e617c76-3e92-4f95-81b0-e0a9977f38a1" containerName="payload-pre-processing" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540553 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6250006e-8963-4565-91e4-6d09ceadf762" containerName="istio-proxy" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540566 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" containerName="maas-api" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540577 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc27530-aef3-4e8d-83b7-ebc205aa50ec" containerName="payload-processing" Nov 04 01:26:51.540580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540588 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2550f115-5ac3-4059-a467-83b06601cda5" containerName="istio-proxy" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540596 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2d0569-9744-4daf-aa0e-6b06c891a357" containerName="main" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540606 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" containerName="istio-proxy" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540613 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540621 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="481d9419-e346-444c-b319-a38160b62de7" containerName="istio-proxy" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540635 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f907ae-3f89-40f0-b5be-96fba4eff5c1" containerName="istio-proxy" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540645 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="71044d9c-6298-48db-b42b-6aab36d15d4d" containerName="istio-proxy" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540657 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="638bc5b7-a66e-43d9-8053-b754b0b00afc" containerName="istio-proxy" Nov 04 01:26:51.541303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.540668 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="33dee613-4f76-4d13-9700-22a0d178c024" containerName="istio-proxy" Nov 04 01:26:51.545185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.545156 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.549535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.549163 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d2e4e3-dockercfg-2rs5w\"" Nov 04 01:26:51.549535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.549241 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d2e4e3\"" Nov 04 01:26:51.550238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.550211 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd"] Nov 04 01:26:51.565072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.565042 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m"] Nov 04 01:26:51.565959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.565937 4312 cpu_manager.go:414] "containerMap: removing container" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" Nov 04 01:26:51.565959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.565959 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" Nov 04 01:26:51.566141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.566121 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5168ab-db6a-423c-81e8-80011c9d13f9" containerName="istio-proxy" Nov 04 01:26:51.569765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.569745 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:51.581706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.581680 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m"] Nov 04 01:26:51.585255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.584895 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-kube-api-access-2bm46\" (UniqueName: \"kubernetes.io/projected/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-kube-api-access-2bm46\") pod \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " Nov 04 01:26:51.585255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.585164 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-maas-api-tls\") pod \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " Nov 04 01:26:51.585255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.585225 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-metrics-certs\") pod \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " Nov 04 01:26:51.585483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.585465 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-openshift-service-ca\") pod \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\" (UID: \"f26e19f6-a465-45b6-85a5-1d2b3b6d68f5\") " Nov 04 01:26:51.586001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.585954 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-openshift-service-ca" pod "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" (UID: "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:51.586434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.586412 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:51.587748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.587722 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-kube-api-access-2bm46" pod "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" (UID: "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5"). InnerVolumeSpecName "kube-api-access-2bm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:51.588519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.588494 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-metrics-certs" pod "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" (UID: "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:51.589506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.589470 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-maas-api-tls" pod "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" (UID: "f26e19f6-a465-45b6-85a5-1d2b3b6d68f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:51.701411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.695488 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d9433c-03e4-4a44-889f-bed3f3c8a24c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.701411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.695731 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbrx\" (UniqueName: \"kubernetes.io/projected/6d423e64-8112-41ed-a035-5f15b34cc2fe-kube-api-access-lsbrx\") pod \"payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:51.701411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.695778 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d423e64-8112-41ed-a035-5f15b34cc2fe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:51.701411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.695870 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2dt\" (UniqueName: \"kubernetes.io/projected/03d9433c-03e4-4a44-889f-bed3f3c8a24c-kube-api-access-mg2dt\") pod \"payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.701411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.696023 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bm46\" (UniqueName: \"kubernetes.io/projected/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-kube-api-access-2bm46\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:51.701411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.696042 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:51.701411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.696059 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:51.803830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.798556 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2dt\" (UniqueName: \"kubernetes.io/projected/03d9433c-03e4-4a44-889f-bed3f3c8a24c-kube-api-access-mg2dt\") pod \"payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.803830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.798755 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d9433c-03e4-4a44-889f-bed3f3c8a24c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.803830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.798963 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbrx\" (UniqueName: \"kubernetes.io/projected/6d423e64-8112-41ed-a035-5f15b34cc2fe-kube-api-access-lsbrx\") pod \"payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:51.803830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.799009 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d423e64-8112-41ed-a035-5f15b34cc2fe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:51.806233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.804321 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d9433c-03e4-4a44-889f-bed3f3c8a24c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.836355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.820570 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d423e64-8112-41ed-a035-5f15b34cc2fe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:51.836355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.831200 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2dt\" (UniqueName: \"kubernetes.io/projected/03d9433c-03e4-4a44-889f-bed3f3c8a24c-kube-api-access-mg2dt\") pod \"payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.838390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.838359 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbrx\" (UniqueName: \"kubernetes.io/projected/6d423e64-8112-41ed-a035-5f15b34cc2fe-kube-api-access-lsbrx\") pod \"payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:51.879582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.875677 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:51.903421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:51.901362 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:52.114521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.114486 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481d9419-e346-444c-b319-a38160b62de7" path="/var/lib/kubelet/pods/481d9419-e346-444c-b319-a38160b62de7/volumes" Nov 04 01:26:52.115179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.115162 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8860371f-a537-41c7-ae99-7d5c165baed9" path="/var/lib/kubelet/pods/8860371f-a537-41c7-ae99-7d5c165baed9/volumes" Nov 04 01:26:52.292885 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:26:52.292853 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d9433c_03e4_4a44_889f_bed3f3c8a24c.slice/crio-f17fe76a48e9d4c83d8dc7501d876b9836704fc032a95ad17685d448846e497b WatchSource:0}: Error finding container f17fe76a48e9d4c83d8dc7501d876b9836704fc032a95ad17685d448846e497b: Status 404 returned error can't find the container with id f17fe76a48e9d4c83d8dc7501d876b9836704fc032a95ad17685d448846e497b Nov 04 01:26:52.293222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.293195 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd"] Nov 04 01:26:52.294391 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:26:52.294326 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d423e64_8112_41ed_a035_5f15b34cc2fe.slice/crio-a4ed1a6a16eab63fe3d8f92e7acc9675f2a2519bed6dd644423c7227b31dd0b3 WatchSource:0}: Error finding container a4ed1a6a16eab63fe3d8f92e7acc9675f2a2519bed6dd644423c7227b31dd0b3: Status 404 returned error can't find the container with id a4ed1a6a16eab63fe3d8f92e7acc9675f2a2519bed6dd644423c7227b31dd0b3 Nov 04 01:26:52.298879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.298853 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m"] Nov 04 01:26:52.458745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.458710 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms"] Nov 04 01:26:52.463782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.463758 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:26:52.464003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.463862 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.464003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.463996 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" event={"ID":"03d9433c-03e4-4a44-889f-bed3f3c8a24c","Type":"ContainerStarted","Data":"eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4"} Nov 04 01:26:52.464189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.464026 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" event={"ID":"03d9433c-03e4-4a44-889f-bed3f3c8a24c","Type":"ContainerStarted","Data":"f17fe76a48e9d4c83d8dc7501d876b9836704fc032a95ad17685d448846e497b"} Nov 04 01:26:52.465099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.465075 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" event={"ID":"6d423e64-8112-41ed-a035-5f15b34cc2fe","Type":"ContainerStarted","Data":"40389157424c2702caf1dd2ca5ca53b5e2049b63d323ee196eed73096a810306"} Nov 04 01:26:52.465200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.465122 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" event={"ID":"6d423e64-8112-41ed-a035-5f15b34cc2fe","Type":"ContainerStarted","Data":"a4ed1a6a16eab63fe3d8f92e7acc9675f2a2519bed6dd644423c7227b31dd0b3"} Nov 04 01:26:52.465200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.465175 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:26:52.465297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.465243 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx" Nov 04 01:26:52.469612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.469592 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-26f3d2\"" Nov 04 01:26:52.469731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.469662 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-26f3d2-dockercfg-qk9ct\"" Nov 04 01:26:52.472124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.472099 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms"] Nov 04 01:26:52.496808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.496749 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" podStartSLOduration=1.496729857 podStartE2EDuration="1.496729857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:52.48530431 +0000 UTC m=+1444.508236331" watchObservedRunningTime="2026-11-04 01:26:52.496729857 +0000 UTC m=+1444.519661868" Nov 04 01:26:52.502222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.502194 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t"] Nov 04 01:26:52.507097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.507078 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:52.508845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.508820 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx"] Nov 04 01:26:52.516575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.516550 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t"] Nov 04 01:26:52.522400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.522374 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6e609c-d7477dd4-88fqx"] Nov 04 01:26:52.548424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.548305 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" podStartSLOduration=1.548291753 podStartE2EDuration="1.548291753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:52.547471327 +0000 UTC m=+1444.570403356" watchObservedRunningTime="2026-11-04 01:26:52.548291753 +0000 UTC m=+1444.571223760" Nov 04 01:26:52.625834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.625797 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5h62\" (UniqueName: \"kubernetes.io/projected/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-kube-api-access-p5h62\") pod \"payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:52.626006 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.625935 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ww42\" (UniqueName: \"kubernetes.io/projected/51241f78-295d-4d43-8ed0-dab0ba0e7503-kube-api-access-6ww42\") pod \"payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.626120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.626102 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-plugins-config-volume\") pod \"payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:52.626291 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.626275 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51241f78-295d-4d43-8ed0-dab0ba0e7503-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.729569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.729517 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ww42\" (UniqueName: \"kubernetes.io/projected/51241f78-295d-4d43-8ed0-dab0ba0e7503-kube-api-access-6ww42\") pod \"payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.730328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.730287 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-plugins-config-volume\") pod \"payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:52.730575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.730552 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51241f78-295d-4d43-8ed0-dab0ba0e7503-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.730731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.730715 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5h62\" (UniqueName: \"kubernetes.io/projected/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-kube-api-access-p5h62\") pod \"payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:52.731959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.731931 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-plugins-config-volume\") pod \"payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:52.743423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.732867 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51241f78-295d-4d43-8ed0-dab0ba0e7503-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.788357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.778441 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ww42\" (UniqueName: \"kubernetes.io/projected/51241f78-295d-4d43-8ed0-dab0ba0e7503-kube-api-access-6ww42\") pod \"payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.792083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.792052 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:52.793990 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.793930 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5h62\" (UniqueName: \"kubernetes.io/projected/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-kube-api-access-p5h62\") pod \"payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:52.827949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:52.827656 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:53.341978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.341947 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t"] Nov 04 01:26:53.342819 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:26:53.342795 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2138bbdb_3ef7_4ad4_83d5_ac22fabcd759.slice/crio-2b9697345d21b4ce2060eff9654cef3627d817211d197d643f8288be73001a08 WatchSource:0}: Error finding container 2b9697345d21b4ce2060eff9654cef3627d817211d197d643f8288be73001a08: Status 404 returned error can't find the container with id 2b9697345d21b4ce2060eff9654cef3627d817211d197d643f8288be73001a08 Nov 04 01:26:53.370868 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:26:53.370842 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51241f78_295d_4d43_8ed0_dab0ba0e7503.slice/crio-e557285a53433582ccda717f2d5e77bde847123c324d0e9d1cd9b0c24f64a2e5 WatchSource:0}: Error finding container e557285a53433582ccda717f2d5e77bde847123c324d0e9d1cd9b0c24f64a2e5: Status 404 returned error can't find the container with id e557285a53433582ccda717f2d5e77bde847123c324d0e9d1cd9b0c24f64a2e5 Nov 04 01:26:53.371855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.371832 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms"] Nov 04 01:26:53.475528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.475481 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" event={"ID":"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759","Type":"ContainerStarted","Data":"981db677fb7d4f70d270aa98af9fd5caedb988b1f3aec3fd70ca3d323ebf7e6e"} Nov 04 01:26:53.475962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.475554 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" event={"ID":"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759","Type":"ContainerStarted","Data":"2b9697345d21b4ce2060eff9654cef3627d817211d197d643f8288be73001a08"} Nov 04 01:26:53.475962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.475583 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:26:53.477514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.477488 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" event={"ID":"51241f78-295d-4d43-8ed0-dab0ba0e7503","Type":"ContainerStarted","Data":"4c213262b658b7f6baa964655b4f2ba2b35d2eefd9447d889aabc426478adcb5"} Nov 04 01:26:53.477647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.477547 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" event={"ID":"51241f78-295d-4d43-8ed0-dab0ba0e7503","Type":"ContainerStarted","Data":"e557285a53433582ccda717f2d5e77bde847123c324d0e9d1cd9b0c24f64a2e5"} Nov 04 01:26:53.477830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.477809 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:26:53.497592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.497548 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" podStartSLOduration=1.4975341229999999 podStartE2EDuration="1.497534123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:53.492797297 +0000 UTC m=+1445.515729317" watchObservedRunningTime="2026-11-04 01:26:53.497534123 +0000 UTC m=+1445.520466130" Nov 04 01:26:53.512863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:53.512815 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" podStartSLOduration=1.512801712 podStartE2EDuration="1.512801712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:53.510593611 +0000 UTC m=+1445.533525636" watchObservedRunningTime="2026-11-04 01:26:53.512801712 +0000 UTC m=+1445.535733719" Nov 04 01:26:54.112791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.112759 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26e19f6-a465-45b6-85a5-1d2b3b6d68f5" path="/var/lib/kubelet/pods/f26e19f6-a465-45b6-85a5-1d2b3b6d68f5/volumes" Nov 04 01:26:54.493381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.492777 4312 generic.go:356] "Generic (PLEG): container finished" podID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerID="a75542098305c2347d93f520d4a090c0102febe9015b5804640881e0b107a92f" exitCode=0 Nov 04 01:26:54.493381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.492943 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" event={"ID":"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e","Type":"ContainerDied","Data":"a75542098305c2347d93f520d4a090c0102febe9015b5804640881e0b107a92f"} Nov 04 01:26:54.495817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.495788 4312 generic.go:356] "Generic (PLEG): container finished" podID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerID="17fde16744070272d4ef9a459aa86054c9299ebf6e439a9882907f463c4b9f6d" exitCode=0 Nov 04 01:26:54.496234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.495985 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" event={"ID":"0136259d-c35d-4e2d-98b9-8e1c76565d09","Type":"ContainerDied","Data":"17fde16744070272d4ef9a459aa86054c9299ebf6e439a9882907f463c4b9f6d"} Nov 04 01:26:54.498600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.498576 4312 generic.go:356] "Generic (PLEG): container finished" podID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerID="ad60383a04bf84ac66ac187fde586eab6409e4f4e49e877c28221f9539e17598" exitCode=0 Nov 04 01:26:54.499796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.498745 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" event={"ID":"b61b2c5e-921a-441d-a727-a35a0fed31d1","Type":"ContainerDied","Data":"ad60383a04bf84ac66ac187fde586eab6409e4f4e49e877c28221f9539e17598"} Nov 04 01:26:54.581501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.581471 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:26:54.624743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.624362 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:26:54.637668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.637646 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-certs\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.637784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.637768 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-data\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.637928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.637912 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-kube-api-access-bnpcz\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-kube-api-access-bnpcz\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.638024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.637921 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-certs" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.638085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638059 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-socket\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.638140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638097 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-data" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.638140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638113 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-credential-socket\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.638242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638184 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-podinfo\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.638242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638228 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-envoy\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.638375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638356 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-socket" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.638441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638386 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-token\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-token\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.638499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638462 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-credential-socket" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.638714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.638698 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61b2c5e-921a-441d-a727-a35a0fed31d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61b2c5e-921a-441d-a727-a35a0fed31d1-istiod-ca-cert\") pod \"b61b2c5e-921a-441d-a727-a35a0fed31d1\" (UID: \"b61b2c5e-921a-441d-a727-a35a0fed31d1\") " Nov 04 01:26:54.639107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.639049 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61b2c5e-921a-441d-a727-a35a0fed31d1-istiod-ca-cert" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:54.639929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.639747 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61b2c5e-921a-441d-a727-a35a0fed31d1-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.639929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.639773 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.639929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.639787 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.639929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.639800 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.639929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.639814 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.640963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.640939 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-kube-api-access-bnpcz" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "kube-api-access-bnpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:54.641051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.641037 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-envoy" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.641704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.641676 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-token" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:54.642772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.642747 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-podinfo" pod "b61b2c5e-921a-441d-a727-a35a0fed31d1" (UID: "b61b2c5e-921a-441d-a727-a35a0fed31d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:54.649255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.649157 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:26:54.741158 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741123 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-data\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.741324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741165 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-certs\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.741324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741223 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istiod-ca-cert\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.741324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741311 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-token\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-token\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.741569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741376 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-envoy\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.741569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741419 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-kube-api-access-574rj\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-kube-api-access-574rj\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.741569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741467 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-data" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.741569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741482 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-certs\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.741569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741531 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-credential-socket\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.741569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741540 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-certs" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.741569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741561 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-podinfo\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.741912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741619 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-data\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.741912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741755 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0136259d-c35d-4e2d-98b9-8e1c76565d09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0136259d-c35d-4e2d-98b9-8e1c76565d09-istiod-ca-cert\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.741912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741784 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-certs" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.741912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741829 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istiod-ca-cert" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:54.741912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741847 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-token\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.741912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741897 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-socket\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.742203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741978 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-podinfo\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.742203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.741979 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-data" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.742203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.742017 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-kube-api-access-vhnwc\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-kube-api-access-vhnwc\") pod \"0136259d-c35d-4e2d-98b9-8e1c76565d09\" (UID: \"0136259d-c35d-4e2d-98b9-8e1c76565d09\") " Nov 04 01:26:54.742203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.742077 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-socket\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.742203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.742137 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-credential-socket" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.742203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.742149 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-envoy\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.742496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.742258 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-credential-socket\") pod \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\" (UID: \"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e\") " Nov 04 01:26:54.742728 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.742701 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0136259d-c35d-4e2d-98b9-8e1c76565d09-istiod-ca-cert" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:54.742956 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.742930 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-credential-socket" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.743235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743209 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0136259d-c35d-4e2d-98b9-8e1c76565d09-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743283 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743300 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743316 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743345 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743362 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743376 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743391 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743405 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743421 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743435 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743448 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnpcz\" (UniqueName: \"kubernetes.io/projected/b61b2c5e-921a-441d-a727-a35a0fed31d1-kube-api-access-bnpcz\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.743674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743659 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-socket" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.744048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.743840 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-socket" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.744656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.744576 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-podinfo" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:54.745280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.745254 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-kube-api-access-574rj" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "kube-api-access-574rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:54.745390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.745299 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-kube-api-access-vhnwc" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "kube-api-access-vhnwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:54.745679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.745658 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-token" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:54.745851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.745829 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-token" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:54.745922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.745905 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-envoy" pod "0136259d-c35d-4e2d-98b9-8e1c76565d09" (UID: "0136259d-c35d-4e2d-98b9-8e1c76565d09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.746323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.746304 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-envoy" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:54.747027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.747007 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-podinfo" pod "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" (UID: "d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:54.845013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.844978 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845010 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845020 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845032 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhnwc\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-kube-api-access-vhnwc\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845043 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845057 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845069 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845078 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845088 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574rj\" (UniqueName: \"kubernetes.io/projected/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e-kube-api-access-574rj\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:54.845199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:54.845098 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0136259d-c35d-4e2d-98b9-8e1c76565d09-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:55.507465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.507424 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" event={"ID":"d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e","Type":"ContainerDied","Data":"32e2305fecc43f2087df1d722cf17c38e27381305ed84fc8b026747ef8163498"} Nov 04 01:26:55.507945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.507440 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2" Nov 04 01:26:55.507945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.507502 4312 scope.go:122] "RemoveContainer" containerID="a75542098305c2347d93f520d4a090c0102febe9015b5804640881e0b107a92f" Nov 04 01:26:55.509255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.509226 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" Nov 04 01:26:55.509404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.509224 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk" event={"ID":"0136259d-c35d-4e2d-98b9-8e1c76565d09","Type":"ContainerDied","Data":"84521e0756bcf4ab23e3f40329e50593ee734252f500f46f06a5bc9787324520"} Nov 04 01:26:55.511037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.511016 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" Nov 04 01:26:55.511149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.511044 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74" event={"ID":"b61b2c5e-921a-441d-a727-a35a0fed31d1","Type":"ContainerDied","Data":"62a82dfe0855e539b7ee45c4a2922d1dc904c162eb13ae645e3dd5f58c6eec94"} Nov 04 01:26:55.520259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.520241 4312 scope.go:122] "RemoveContainer" containerID="17fde16744070272d4ef9a459aa86054c9299ebf6e439a9882907f463c4b9f6d" Nov 04 01:26:55.532230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.532198 4312 scope.go:122] "RemoveContainer" containerID="ad60383a04bf84ac66ac187fde586eab6409e4f4e49e877c28221f9539e17598" Nov 04 01:26:55.541654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.541631 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74"] Nov 04 01:26:55.549372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.549347 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-8dw74"] Nov 04 01:26:55.559533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.559508 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk"] Nov 04 01:26:55.563276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.563254 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-bqkxk"] Nov 04 01:26:55.577511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.577487 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2"] Nov 04 01:26:55.582428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:55.582404 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-7rsc2"] Nov 04 01:26:56.112166 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:56.112132 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" path="/var/lib/kubelet/pods/0136259d-c35d-4e2d-98b9-8e1c76565d09/volumes" Nov 04 01:26:56.112748 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:56.112730 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" path="/var/lib/kubelet/pods/b61b2c5e-921a-441d-a727-a35a0fed31d1/volumes" Nov 04 01:26:56.113416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:56.113393 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" path="/var/lib/kubelet/pods/d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e/volumes" Nov 04 01:26:57.026733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.026519 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv"] Nov 04 01:26:57.027092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.026805 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" containerID="cri-o://b16842c1e0e285c4ffc53b6b8fc595248f7d2698a1af04bf24348b3a454f410b" gracePeriod=30 Nov 04 01:26:57.045905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.045873 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d"] Nov 04 01:26:57.046139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.046116 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" containerID="cri-o://07fdc2ae4a43cad95b88facbe9c90b77e3ddf606f66763256b31b52fabb0a3ae" gracePeriod=30 Nov 04 01:26:57.158030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.157993 4312 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:26:57.158193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.158059 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:26:57.546465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.546434 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b"] Nov 04 01:26:57.547271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547251 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" Nov 04 01:26:57.547271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547270 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" Nov 04 01:26:57.547459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547284 4312 cpu_manager.go:414] "containerMap: removing container" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" Nov 04 01:26:57.547459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547290 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" Nov 04 01:26:57.547459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547314 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" Nov 04 01:26:57.547459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547319 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" Nov 04 01:26:57.547668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547480 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0136259d-c35d-4e2d-98b9-8e1c76565d09" containerName="istio-proxy" Nov 04 01:26:57.547668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547502 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61b2c5e-921a-441d-a727-a35a0fed31d1" containerName="istio-proxy" Nov 04 01:26:57.547668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.547508 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9dd4b10-4202-4b7b-9759-2e7a9f8b9c5e" containerName="istio-proxy" Nov 04 01:26:57.553071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.553049 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.561553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.561513 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-27bc4f8b-openshift-default-dockercfg-6dxmm\"" Nov 04 01:26:57.564476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.564436 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b"] Nov 04 01:26:57.569865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.569838 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv"] Nov 04 01:26:57.582354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.575076 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.595450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.595416 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv"] Nov 04 01:26:57.644477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.644436 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplgs\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-kube-api-access-gplgs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.644649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.644531 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.644721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.644692 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.644767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.644742 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.644885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.644863 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.644997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.644981 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.645059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.645037 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.645161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.645142 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.645356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.645311 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a9ca41-20ef-4130-bd8c-df377b06c567-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.746892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.746856 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.746905 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfd62be-e511-4d76-be35-a023febdbda2-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.746959 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.746978 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747021 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747122 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a9ca41-20ef-4130-bd8c-df377b06c567-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747172 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplgs\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-kube-api-access-gplgs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747228 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747275 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747288 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747306 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747497 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747548 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747580 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fmw\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-kube-api-access-w8fmw\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747650 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747722 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfd62be-e511-4d76-be35-a023febdbda2-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.747870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747728 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.747870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747835 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.748024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747877 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.748024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747901 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a9ca41-20ef-4130-bd8c-df377b06c567-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.748024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747910 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.748024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.747964 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.748268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.748244 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.749902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.749882 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.750008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.749947 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.758795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.758769 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.759074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.759055 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplgs\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-kube-api-access-gplgs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.849951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.849855 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfd62be-e511-4d76-be35-a023febdbda2-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850022 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850190 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850133 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfd62be-e511-4d76-be35-a023febdbda2-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850215 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850238 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850266 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850462 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850491 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850557 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850596 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850635 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850640 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fmw\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-kube-api-access-w8fmw\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.850892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850845 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.851089 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.850945 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfd62be-e511-4d76-be35-a023febdbda2-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.852675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.852647 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.852753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.852654 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfd62be-e511-4d76-be35-a023febdbda2-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.862280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.862234 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.862408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.862387 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fmw\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-kube-api-access-w8fmw\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:57.894410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.884040 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:57.939650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:57.939355 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:58.319129 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:26:58.319095 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a9ca41_20ef_4130_bd8c_df377b06c567.slice/crio-069d37feea5113689b0c1b5f9fe31d288a80e0f16610139d326ee97e51c4cb78 WatchSource:0}: Error finding container 069d37feea5113689b0c1b5f9fe31d288a80e0f16610139d326ee97e51c4cb78: Status 404 returned error can't find the container with id 069d37feea5113689b0c1b5f9fe31d288a80e0f16610139d326ee97e51c4cb78 Nov 04 01:26:58.320602 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:26:58.320573 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfd62be_e511_4d76_be35_a023febdbda2.slice/crio-d4ea585cdb5e831df09249bae2b834d1cb6265ce26f5f7d2b33d03d2344b4998 WatchSource:0}: Error finding container d4ea585cdb5e831df09249bae2b834d1cb6265ce26f5f7d2b33d03d2344b4998: Status 404 returned error can't find the container with id d4ea585cdb5e831df09249bae2b834d1cb6265ce26f5f7d2b33d03d2344b4998 Nov 04 01:26:58.322415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.322391 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv"] Nov 04 01:26:58.322640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.322608 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:26:58.322695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.322682 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:26:58.322733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.322714 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:26:58.323617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.323574 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:26:58.323720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.323657 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:26:58.323720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.323699 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:26:58.328206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.328187 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b"] Nov 04 01:26:58.539884 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.539568 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" event={"ID":"7dfd62be-e511-4d76-be35-a023febdbda2","Type":"ContainerStarted","Data":"04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa"} Nov 04 01:26:58.539884 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.539649 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" event={"ID":"7dfd62be-e511-4d76-be35-a023febdbda2","Type":"ContainerStarted","Data":"d4ea585cdb5e831df09249bae2b834d1cb6265ce26f5f7d2b33d03d2344b4998"} Nov 04 01:26:58.543135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.542161 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" event={"ID":"33a9ca41-20ef-4130-bd8c-df377b06c567","Type":"ContainerStarted","Data":"49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e"} Nov 04 01:26:58.543135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.542228 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" event={"ID":"33a9ca41-20ef-4130-bd8c-df377b06c567","Type":"ContainerStarted","Data":"069d37feea5113689b0c1b5f9fe31d288a80e0f16610139d326ee97e51c4cb78"} Nov 04 01:26:58.597993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.597880 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podStartSLOduration=1.597859439 podStartE2EDuration="1.597859439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:58.570316609 +0000 UTC m=+1450.593248654" watchObservedRunningTime="2026-11-04 01:26:58.597859439 +0000 UTC m=+1450.620791448" Nov 04 01:26:58.641784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.641719 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podStartSLOduration=1.641699795 podStartE2EDuration="1.641699795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:58.625284707 +0000 UTC m=+1450.648216716" watchObservedRunningTime="2026-11-04 01:26:58.641699795 +0000 UTC m=+1450.664631802" Nov 04 01:26:58.891404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.889619 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:26:58.944712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:58.944624 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:26:59.502595 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:26:59.502527 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:26:59.894399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:59.891080 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:26:59.894399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:59.891154 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 01:26:59.941190 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:59.941156 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:26:59.941396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:26:59.941209 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 01:27:00.885357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:00.884780 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:27:00.885357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:00.884842 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 01:27:00.942743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:00.942705 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:27:00.942910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:00.942766 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 01:27:00.944019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:00.943992 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754"] Nov 04 01:27:00.944438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:00.944409 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" podUID="4304bb0b-2e6b-42c8-abaa-6d13d62519dc" containerName="maas-api" containerID="cri-o://2a8a8d0b167b5974a794ee6fbc61ea9c386bd4cd7ff93f9da5097033b0f5e10e" gracePeriod=30 Nov 04 01:27:01.705037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:01.704884 4312 generic.go:356] "Generic (PLEG): container finished" podID="4304bb0b-2e6b-42c8-abaa-6d13d62519dc" containerID="2a8a8d0b167b5974a794ee6fbc61ea9c386bd4cd7ff93f9da5097033b0f5e10e" exitCode=0 Nov 04 01:27:01.705037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:01.704965 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" event={"ID":"4304bb0b-2e6b-42c8-abaa-6d13d62519dc","Type":"ContainerDied","Data":"2a8a8d0b167b5974a794ee6fbc61ea9c386bd4cd7ff93f9da5097033b0f5e10e"} Nov 04 01:27:01.908505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:01.908461 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:27:01.909008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:01.908560 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 01:27:01.941382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:01.940738 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:27:01.941382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:01.940801 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 01:27:02.013355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.013066 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:27:02.029150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.029120 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-openshift-service-ca\") pod \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " Nov 04 01:27:02.029313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.029284 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-kube-api-access-ftg5k\" (UniqueName: \"kubernetes.io/projected/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-kube-api-access-ftg5k\") pod \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " Nov 04 01:27:02.029393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.029316 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-metrics-certs\") pod \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " Nov 04 01:27:02.029508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.029487 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-openshift-service-ca" pod "4304bb0b-2e6b-42c8-abaa-6d13d62519dc" (UID: "4304bb0b-2e6b-42c8-abaa-6d13d62519dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:02.029579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.029517 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-maas-api-tls\") pod \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\" (UID: \"4304bb0b-2e6b-42c8-abaa-6d13d62519dc\") " Nov 04 01:27:02.030274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.030250 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:02.032049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.032023 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-metrics-certs" pod "4304bb0b-2e6b-42c8-abaa-6d13d62519dc" (UID: "4304bb0b-2e6b-42c8-abaa-6d13d62519dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:02.032272 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.032245 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-kube-api-access-ftg5k" pod "4304bb0b-2e6b-42c8-abaa-6d13d62519dc" (UID: "4304bb0b-2e6b-42c8-abaa-6d13d62519dc"). InnerVolumeSpecName "kube-api-access-ftg5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:02.032586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.032564 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-maas-api-tls" pod "4304bb0b-2e6b-42c8-abaa-6d13d62519dc" (UID: "4304bb0b-2e6b-42c8-abaa-6d13d62519dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:02.139366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.138865 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftg5k\" (UniqueName: \"kubernetes.io/projected/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-kube-api-access-ftg5k\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:02.139366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.138899 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:02.139366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.138914 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304bb0b-2e6b-42c8-abaa-6d13d62519dc-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:02.724501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.724467 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" event={"ID":"4304bb0b-2e6b-42c8-abaa-6d13d62519dc","Type":"ContainerDied","Data":"f06d6f62c5498e5f7d1dde5054186985211e960c85b6bee23c27684e454559ee"} Nov 04 01:27:02.724622 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.724536 4312 scope.go:122] "RemoveContainer" containerID="2a8a8d0b167b5974a794ee6fbc61ea9c386bd4cd7ff93f9da5097033b0f5e10e" Nov 04 01:27:02.724702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.724678 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754" Nov 04 01:27:02.730384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.730301 4312 generic.go:356] "Generic (PLEG): container finished" podID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerID="07fdc2ae4a43cad95b88facbe9c90b77e3ddf606f66763256b31b52fabb0a3ae" exitCode=0 Nov 04 01:27:02.730510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.730420 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" event={"ID":"9ecd9d73-fcf8-4b10-9ed4-02edf428baee","Type":"ContainerDied","Data":"07fdc2ae4a43cad95b88facbe9c90b77e3ddf606f66763256b31b52fabb0a3ae"} Nov 04 01:27:02.746320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.746156 4312 generic.go:356] "Generic (PLEG): container finished" podID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerID="b16842c1e0e285c4ffc53b6b8fc595248f7d2698a1af04bf24348b3a454f410b" exitCode=0 Nov 04 01:27:02.746320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.746229 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" event={"ID":"4347a44e-f700-4c34-8bb8-4f877ddaf5a2","Type":"ContainerDied","Data":"b16842c1e0e285c4ffc53b6b8fc595248f7d2698a1af04bf24348b3a454f410b"} Nov 04 01:27:02.805989 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.804119 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754"] Nov 04 01:27:02.814518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.814492 4312 scope.go:122] "RemoveContainer" containerID="034b430f98e3680852a0768fbe012b964fff860e691c7ce442d5f7aac13f3d0b" Nov 04 01:27:02.819682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.819656 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fdd516-65f77dbf6-gd754"] Nov 04 01:27:02.826589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.826324 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:27:02.909584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.908678 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:27:02.909584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.908733 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 01:27:02.924161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.924131 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:27:02.929216 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929184 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-envoy\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929266 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-podinfo\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929352 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-kube-api-access-jd2b9\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-kube-api-access-jd2b9\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.929427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929402 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-token\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929447 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-kube-api-access-cws2m\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-kube-api-access-cws2m\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929481 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-socket\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929517 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istiod-ca-cert\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.929569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929552 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-certs\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.929603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929589 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istiod-ca-cert\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929659 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-credential-socket\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.929735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929720 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-credential-socket\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929811 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-certs\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.929943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929927 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-data\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.930002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.929989 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-socket\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.930089 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.930073 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-data\") pod \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\" (UID: \"9ecd9d73-fcf8-4b10-9ed4-02edf428baee\") " Nov 04 01:27:02.930123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.930113 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-token\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-token\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.930219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.930203 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-podinfo\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.930268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.930254 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-envoy\") pod \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\" (UID: \"4347a44e-f700-4c34-8bb8-4f877ddaf5a2\") " Nov 04 01:27:02.931768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.931592 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istiod-ca-cert" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:02.933781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.932827 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-certs" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.933781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.933041 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-socket" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.933781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.933225 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-data" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.933781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.933439 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-credential-socket" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.933781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.933614 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-credential-socket" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.936444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.936403 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-socket" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.936537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.936483 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-certs" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.936610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.936567 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istiod-ca-cert" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:02.936740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.936717 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-data" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.937131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.937091 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-envoy" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.940957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.940376 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-envoy" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:02.940957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.940768 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-podinfo" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:27:02.940957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.940920 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-kube-api-access-jd2b9" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "kube-api-access-jd2b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:02.941683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.941231 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:27:02.941683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.941287 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 01:27:02.941683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.941647 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-podinfo" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:27:02.942068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.942009 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-token" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:02.942424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.942240 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-token" pod "4347a44e-f700-4c34-8bb8-4f877ddaf5a2" (UID: "4347a44e-f700-4c34-8bb8-4f877ddaf5a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:02.942424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:02.942320 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-kube-api-access-cws2m" pod "9ecd9d73-fcf8-4b10-9ed4-02edf428baee" (UID: "9ecd9d73-fcf8-4b10-9ed4-02edf428baee"). InnerVolumeSpecName "kube-api-access-cws2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032353 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032386 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032396 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032405 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032416 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032424 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032438 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032451 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032460 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2b9\" (UniqueName: \"kubernetes.io/projected/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-kube-api-access-jd2b9\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032469 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032478 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cws2m\" (UniqueName: \"kubernetes.io/projected/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-kube-api-access-cws2m\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032488 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032496 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032505 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032518 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032529 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4347a44e-f700-4c34-8bb8-4f877ddaf5a2-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032537 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.038011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.032546 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd9d73-fcf8-4b10-9ed4-02edf428baee-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:03.491405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.490585 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:27:03.503359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.500457 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:27:03.823666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.821627 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" Nov 04 01:27:03.823666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.821620 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d" event={"ID":"9ecd9d73-fcf8-4b10-9ed4-02edf428baee","Type":"ContainerDied","Data":"d14458bfd26684d2cf76bbd23f2f06362dfb4c84d1d36d7c3e467dfb72f683a7"} Nov 04 01:27:03.823666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.823423 4312 scope.go:122] "RemoveContainer" containerID="07fdc2ae4a43cad95b88facbe9c90b77e3ddf606f66763256b31b52fabb0a3ae" Nov 04 01:27:03.825802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.824629 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" Nov 04 01:27:03.827349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.824644 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv" event={"ID":"4347a44e-f700-4c34-8bb8-4f877ddaf5a2","Type":"ContainerDied","Data":"3eb7cfd16e53e6c88557362536835fd353c77e514e9e0f95edfb5f81d01ae001"} Nov 04 01:27:03.831734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.831713 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp"] Nov 04 01:27:03.920781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.899654 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:27:03.920781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.899710 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 01:27:03.930627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.930459 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d"] Nov 04 01:27:03.932783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.932561 4312 scope.go:122] "RemoveContainer" containerID="b16842c1e0e285c4ffc53b6b8fc595248f7d2698a1af04bf24348b3a454f410b" Nov 04 01:27:03.934642 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.934589 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-jxq9d"] Nov 04 01:27:03.940132 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.940103 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:27:03.940219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.940147 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 01:27:03.940495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.940475 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv"] Nov 04 01:27:03.944311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:03.944286 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-bktzv"] Nov 04 01:27:04.118146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.116997 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4304bb0b-2e6b-42c8-abaa-6d13d62519dc" path="/var/lib/kubelet/pods/4304bb0b-2e6b-42c8-abaa-6d13d62519dc/volumes" Nov 04 01:27:04.118146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.117794 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" path="/var/lib/kubelet/pods/4347a44e-f700-4c34-8bb8-4f877ddaf5a2/volumes" Nov 04 01:27:04.118449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.118410 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" path="/var/lib/kubelet/pods/9ecd9d73-fcf8-4b10-9ed4-02edf428baee/volumes" Nov 04 01:27:04.502975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.502897 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:27:04.504380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.504358 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:27:04.947360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.946470 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 01:27:04.947360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.946541 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 01:27:04.959729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.957922 4312 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 01:27:04.959729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:04.957975 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 01:27:05.229046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.228979 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz"] Nov 04 01:27:05.229886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.229865 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" Nov 04 01:27:05.229982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.229887 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" Nov 04 01:27:05.229982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.229900 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4304bb0b-2e6b-42c8-abaa-6d13d62519dc" containerName="maas-api" Nov 04 01:27:05.229982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.229905 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4304bb0b-2e6b-42c8-abaa-6d13d62519dc" containerName="maas-api" Nov 04 01:27:05.229982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.229943 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" Nov 04 01:27:05.229982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.229949 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" Nov 04 01:27:05.230135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.230064 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" Nov 04 01:27:05.230135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.230077 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" Nov 04 01:27:05.230135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.230085 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4304bb0b-2e6b-42c8-abaa-6d13d62519dc" containerName="maas-api" Nov 04 01:27:05.230135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.230095 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4347a44e-f700-4c34-8bb8-4f877ddaf5a2" containerName="istio-proxy" Nov 04 01:27:05.269791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.269754 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz"] Nov 04 01:27:05.269953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.269879 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.276222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.274784 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d2e4e3-serving-cert\"" Nov 04 01:27:05.339867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.339825 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-maas-api-tls\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.340156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.339904 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjft\" (UniqueName: \"kubernetes.io/projected/64548438-f3dd-488f-a623-9cd17e97af8e-kube-api-access-vgjft\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.340156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.340015 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64548438-f3dd-488f-a623-9cd17e97af8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.340306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.340245 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-metrics-certs\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.452408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.449501 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-maas-api-tls\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.452408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.449579 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjft\" (UniqueName: \"kubernetes.io/projected/64548438-f3dd-488f-a623-9cd17e97af8e-kube-api-access-vgjft\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.452408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.449630 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64548438-f3dd-488f-a623-9cd17e97af8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.452408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.449835 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-metrics-certs\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.462369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.455243 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-maas-api-tls\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.462369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.455938 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64548438-f3dd-488f-a623-9cd17e97af8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.491447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.486062 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-metrics-certs\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.491447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.486955 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjft\" (UniqueName: \"kubernetes.io/projected/64548438-f3dd-488f-a623-9cd17e97af8e-kube-api-access-vgjft\") pod \"maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.589581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.589545 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:05.929411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.926358 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:27:05.929411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.928027 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:27:05.930168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.930115 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:27:05.956764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.946777 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz"] Nov 04 01:27:05.963065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.963045 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:27:05.963387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.963369 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:27:05.964194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.964177 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:27:05.968518 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:05.968496 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64548438_f3dd_488f_a623_9cd17e97af8e.slice/crio-6f405901d844f3445e342f72dc6d072ae1266adfe3f6690fa4292f20bd8ddd5a WatchSource:0}: Error finding container 6f405901d844f3445e342f72dc6d072ae1266adfe3f6690fa4292f20bd8ddd5a: Status 404 returned error can't find the container with id 6f405901d844f3445e342f72dc6d072ae1266adfe3f6690fa4292f20bd8ddd5a Nov 04 01:27:05.970737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:05.970718 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:27:06.033998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.033954 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" event={"ID":"64548438-f3dd-488f-a623-9cd17e97af8e","Type":"ContainerStarted","Data":"6f405901d844f3445e342f72dc6d072ae1266adfe3f6690fa4292f20bd8ddd5a"} Nov 04 01:27:06.123243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.123210 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z"] Nov 04 01:27:06.124639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.124617 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" Nov 04 01:27:06.124739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.124642 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ecd9d73-fcf8-4b10-9ed4-02edf428baee" containerName="istio-proxy" Nov 04 01:27:06.137238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.135617 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.137238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.136769 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z"] Nov 04 01:27:06.142657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.142622 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-26f3d2-serving-cert\"" Nov 04 01:27:06.224917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.224823 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b81fcbb-dfaf-467c-9da0-913c1f77a907-openshift-service-ca\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.224917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.224890 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-metrics-certs\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.225187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.225012 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfljh\" (UniqueName: \"kubernetes.io/projected/2b81fcbb-dfaf-467c-9da0-913c1f77a907-kube-api-access-rfljh\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.225187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.225154 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-maas-api-tls\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.353361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.349633 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-maas-api-tls\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.353361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.349763 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b81fcbb-dfaf-467c-9da0-913c1f77a907-openshift-service-ca\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.353361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.349861 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-metrics-certs\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.353361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.349938 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfljh\" (UniqueName: \"kubernetes.io/projected/2b81fcbb-dfaf-467c-9da0-913c1f77a907-kube-api-access-rfljh\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.405290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.404197 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfljh\" (UniqueName: \"kubernetes.io/projected/2b81fcbb-dfaf-467c-9da0-913c1f77a907-kube-api-access-rfljh\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.405290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.404917 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b81fcbb-dfaf-467c-9da0-913c1f77a907-openshift-service-ca\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.428426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.428390 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-maas-api-tls\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.449066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.449033 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-metrics-certs\") pod \"maas-api-e2e-api-a-26f3d2-847994b4db-lf26z\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.466994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.466895 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:06.880454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:06.880426 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z"] Nov 04 01:27:06.880570 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:06.880523 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b81fcbb_dfaf_467c_9da0_913c1f77a907.slice/crio-bfaeaf2a2fa64d992114919a5dd53180fe5cd18e7d0122eb93ee8c2f11937943 WatchSource:0}: Error finding container bfaeaf2a2fa64d992114919a5dd53180fe5cd18e7d0122eb93ee8c2f11937943: Status 404 returned error can't find the container with id bfaeaf2a2fa64d992114919a5dd53180fe5cd18e7d0122eb93ee8c2f11937943 Nov 04 01:27:07.175202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:07.175164 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" event={"ID":"64548438-f3dd-488f-a623-9cd17e97af8e","Type":"ContainerStarted","Data":"70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90"} Nov 04 01:27:07.176025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:07.176001 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:07.425390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:07.407916 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" event={"ID":"2b81fcbb-dfaf-467c-9da0-913c1f77a907","Type":"ContainerStarted","Data":"bfaeaf2a2fa64d992114919a5dd53180fe5cd18e7d0122eb93ee8c2f11937943"} Nov 04 01:27:08.558507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.558403 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" event={"ID":"2b81fcbb-dfaf-467c-9da0-913c1f77a907","Type":"ContainerStarted","Data":"a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f"} Nov 04 01:27:08.559080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.559059 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:08.596360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.590399 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" podStartSLOduration=2.9724416160000002 podStartE2EDuration="3.590376616s" totalImagesPullingTime="617.935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:07.295246278 +0000 UTC m=+1459.318178297" watchObservedRunningTime="2026-11-04 01:27:08.590376616 +0000 UTC m=+1460.613308625" Nov 04 01:27:08.614354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.608163 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" podStartSLOduration=2.046698165 podStartE2EDuration="2.608144272s" totalImagesPullingTime="561.446107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:08.587841723 +0000 UTC m=+1460.610773731" watchObservedRunningTime="2026-11-04 01:27:08.608144272 +0000 UTC m=+1460.631076283" Nov 04 01:27:08.803731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.803318 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv"] Nov 04 01:27:08.803731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.803579 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" containerID="cri-o://04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa" gracePeriod=30 Nov 04 01:27:08.814454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.814386 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b"] Nov 04 01:27:08.814624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.814597 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" containerID="cri-o://49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e" gracePeriod=30 Nov 04 01:27:08.834694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:08.834660 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" podUID="e3901502-f9b2-4a22-aba1-43a7947e7047" containerName="payload-pre-processing" containerID="cri-o://fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904" gracePeriod=25 Nov 04 01:27:09.277955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.277927 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:27:09.440920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.440817 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3901502-f9b2-4a22-aba1-43a7947e7047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3901502-f9b2-4a22-aba1-43a7947e7047-plugins-config-volume\") pod \"e3901502-f9b2-4a22-aba1-43a7947e7047\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " Nov 04 01:27:09.441084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.440925 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3901502-f9b2-4a22-aba1-43a7947e7047-kube-api-access-qvwxt\" (UniqueName: \"kubernetes.io/projected/e3901502-f9b2-4a22-aba1-43a7947e7047-kube-api-access-qvwxt\") pod \"e3901502-f9b2-4a22-aba1-43a7947e7047\" (UID: \"e3901502-f9b2-4a22-aba1-43a7947e7047\") " Nov 04 01:27:09.441175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.441151 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3901502-f9b2-4a22-aba1-43a7947e7047-plugins-config-volume" pod "e3901502-f9b2-4a22-aba1-43a7947e7047" (UID: "e3901502-f9b2-4a22-aba1-43a7947e7047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:09.441630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.441609 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3901502-f9b2-4a22-aba1-43a7947e7047-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:09.442921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.442891 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3901502-f9b2-4a22-aba1-43a7947e7047-kube-api-access-qvwxt" pod "e3901502-f9b2-4a22-aba1-43a7947e7047" (UID: "e3901502-f9b2-4a22-aba1-43a7947e7047"). InnerVolumeSpecName "kube-api-access-qvwxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:09.547359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.546352 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvwxt\" (UniqueName: \"kubernetes.io/projected/e3901502-f9b2-4a22-aba1-43a7947e7047-kube-api-access-qvwxt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:09.599853 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.599817 4312 generic.go:356] "Generic (PLEG): container finished" podID="e3901502-f9b2-4a22-aba1-43a7947e7047" containerID="fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904" exitCode=0 Nov 04 01:27:09.600297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.599867 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" Nov 04 01:27:09.600297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.599906 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" event={"ID":"e3901502-f9b2-4a22-aba1-43a7947e7047","Type":"ContainerDied","Data":"fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904"} Nov 04 01:27:09.600297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.599977 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp" event={"ID":"e3901502-f9b2-4a22-aba1-43a7947e7047","Type":"ContainerDied","Data":"4cc63784325e519d091294d404404be1cade447ba25157e94cab2f03e8a75dd0"} Nov 04 01:27:09.600297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.600005 4312 scope.go:122] "RemoveContainer" containerID="fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904" Nov 04 01:27:09.611738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.611720 4312 scope.go:122] "RemoveContainer" containerID="fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904" Nov 04 01:27:09.612048 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:09.612028 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904\": container with ID starting with fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904 not found: ID does not exist" containerID="fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904" Nov 04 01:27:09.612118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.612056 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904"} err="failed to get container status \"fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904\": rpc error: code = NotFound desc = could not find container \"fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904\": container with ID starting with fdbc9965a973c7e68a0d4bad65e5d056c243d574946580d1437f99e92deec904 not found: ID does not exist" Nov 04 01:27:09.637501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.637470 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp"] Nov 04 01:27:09.641480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:09.641459 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-20a250-6c68c6bc47-2kkrp"] Nov 04 01:27:10.111786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.111753 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3901502-f9b2-4a22-aba1-43a7947e7047" path="/var/lib/kubelet/pods/e3901502-f9b2-4a22-aba1-43a7947e7047/volumes" Nov 04 01:27:10.136201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.136165 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5"] Nov 04 01:27:10.137376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.137329 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e3901502-f9b2-4a22-aba1-43a7947e7047" containerName="payload-pre-processing" Nov 04 01:27:10.137376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.137368 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3901502-f9b2-4a22-aba1-43a7947e7047" containerName="payload-pre-processing" Nov 04 01:27:10.137568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.137525 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3901502-f9b2-4a22-aba1-43a7947e7047" containerName="payload-pre-processing" Nov 04 01:27:10.143774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.143750 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.161653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.161619 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-695ce9-openshift-default-dockercfg-xzhcj\"" Nov 04 01:27:10.170522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.170493 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5"] Nov 04 01:27:10.262508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.262470 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-token\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.262708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.262523 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istiod-ca-cert\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.262708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.262678 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-certs\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.262839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.262732 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-socket\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.262839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.262772 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxf5j\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-kube-api-access-vxf5j\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.262839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.262789 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-podinfo\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.263017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.262996 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-credential-socket\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.263121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.263107 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-envoy\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.263177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.263145 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-data\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.391460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.391370 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-data\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.391785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.391763 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-token\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.391943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.391927 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istiod-ca-cert\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.392103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.392088 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-certs\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.392235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.392221 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-socket\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.392389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.392369 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxf5j\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-kube-api-access-vxf5j\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.392499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.392484 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-podinfo\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.392726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.392705 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-credential-socket\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.392923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.392908 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-envoy\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.418501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.418466 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-data\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.440504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.418481 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-certs\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.440730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.423355 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-socket\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.440862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.423647 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-credential-socket\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.440970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.423788 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-envoy\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.441075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.426133 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-podinfo\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.445354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.441837 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istiod-ca-cert\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.468149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.449167 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-token\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.514592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.514024 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxf5j\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-kube-api-access-vxf5j\") pod \"e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:10.775240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:10.775150 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:11.187110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.187078 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5"] Nov 04 01:27:11.192020 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:11.191992 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdd5980_cd0c_4f7c_9e15_4b2b6d828910.slice/crio-e7e9722abeec9e9b83fc1a6e15e0210cfc3905caa5f786a1f54b6a73abd4ee6a WatchSource:0}: Error finding container e7e9722abeec9e9b83fc1a6e15e0210cfc3905caa5f786a1f54b6a73abd4ee6a: Status 404 returned error can't find the container with id e7e9722abeec9e9b83fc1a6e15e0210cfc3905caa5f786a1f54b6a73abd4ee6a Nov 04 01:27:11.195174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.195134 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:11.195250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.195223 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:11.195285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.195256 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:11.747319 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.747284 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" event={"ID":"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910","Type":"ContainerStarted","Data":"86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b"} Nov 04 01:27:11.747521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.747384 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" event={"ID":"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910","Type":"ContainerStarted","Data":"e7e9722abeec9e9b83fc1a6e15e0210cfc3905caa5f786a1f54b6a73abd4ee6a"} Nov 04 01:27:11.774165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.774119 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podStartSLOduration=1.7741052339999999 podStartE2EDuration="1.774105234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:11.77153524 +0000 UTC m=+1463.794467265" watchObservedRunningTime="2026-11-04 01:27:11.774105234 +0000 UTC m=+1463.797037240" Nov 04 01:27:11.775269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:11.775245 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:12.781997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:12.781970 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:13.567703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:13.567675 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:27:13.763050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:13.763022 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:13.764030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:13.764011 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:27:14.148000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.147978 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:27:14.151719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.151694 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:27:14.294574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294537 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-credential-socket\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.294574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294575 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a9ca41-20ef-4130-bd8c-df377b06c567-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a9ca41-20ef-4130-bd8c-df377b06c567-istiod-ca-cert\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.294839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294601 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-envoy\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.294839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294618 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfd62be-e511-4d76-be35-a023febdbda2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfd62be-e511-4d76-be35-a023febdbda2-istiod-ca-cert\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.294839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294652 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-certs\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.294839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294745 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dfd62be-e511-4d76-be35-a023febdbda2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfd62be-e511-4d76-be35-a023febdbda2-istio-podinfo\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.294839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294823 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-certs\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.295074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.294883 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-credential-socket" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.295151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295119 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-certs" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.295280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295178 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfd62be-e511-4d76-be35-a023febdbda2-istiod-ca-cert" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:14.295280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295271 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-data\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.295434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295314 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-kube-api-access-gplgs\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-kube-api-access-gplgs\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.295434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295417 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-envoy\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.295542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295450 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-certs" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.295542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295472 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-token\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-token\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.295659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295554 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-data" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.295659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295564 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a9ca41-20ef-4130-bd8c-df377b06c567-istiod-ca-cert" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:14.295659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295555 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-podinfo\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.295821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295737 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-istio-token\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-istio-token\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.295874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295825 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-credential-socket\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.295963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.295946 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-socket\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.296071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.296054 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-data\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.296156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.296138 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-kube-api-access-w8fmw\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-kube-api-access-w8fmw\") pod \"7dfd62be-e511-4d76-be35-a023febdbda2\" (UID: \"7dfd62be-e511-4d76-be35-a023febdbda2\") " Nov 04 01:27:14.296656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.296256 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-socket\") pod \"33a9ca41-20ef-4130-bd8c-df377b06c567\" (UID: \"33a9ca41-20ef-4130-bd8c-df377b06c567\") " Nov 04 01:27:14.296656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.296550 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-socket" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.296656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.296614 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-data" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.297007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.296978 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-socket" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.297205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297179 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-credential-socket" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297730 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297762 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297778 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297793 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297806 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297820 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297833 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297847 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a9ca41-20ef-4130-bd8c-df377b06c567-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297860 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfd62be-e511-4d76-be35-a023febdbda2-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297875 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.297903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.297879 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dfd62be-e511-4d76-be35-a023febdbda2-istio-podinfo" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:27:14.298326 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.298031 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-kube-api-access-gplgs" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "kube-api-access-gplgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:14.299276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.299250 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-envoy" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.299276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.299263 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-podinfo" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:27:14.299467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.299448 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-istio-token" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:14.299641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.299517 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-envoy" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:14.299641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.299517 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-kube-api-access-w8fmw" pod "7dfd62be-e511-4d76-be35-a023febdbda2" (UID: "7dfd62be-e511-4d76-be35-a023febdbda2"). InnerVolumeSpecName "kube-api-access-w8fmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:14.299737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.299689 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-token" pod "33a9ca41-20ef-4130-bd8c-df377b06c567" (UID: "33a9ca41-20ef-4130-bd8c-df377b06c567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414800 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414845 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414862 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414884 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fmw\" (UniqueName: \"kubernetes.io/projected/7dfd62be-e511-4d76-be35-a023febdbda2-kube-api-access-w8fmw\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414902 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a9ca41-20ef-4130-bd8c-df377b06c567-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414915 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfd62be-e511-4d76-be35-a023febdbda2-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414929 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplgs\" (UniqueName: \"kubernetes.io/projected/33a9ca41-20ef-4130-bd8c-df377b06c567-kube-api-access-gplgs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.551405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.414951 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfd62be-e511-4d76-be35-a023febdbda2-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:14.613377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.611691 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:27:14.772486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.772450 4312 generic.go:356] "Generic (PLEG): container finished" podID="7dfd62be-e511-4d76-be35-a023febdbda2" containerID="04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa" exitCode=0 Nov 04 01:27:14.772651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.772524 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" Nov 04 01:27:14.772651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.772544 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" event={"ID":"7dfd62be-e511-4d76-be35-a023febdbda2","Type":"ContainerDied","Data":"04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa"} Nov 04 01:27:14.772651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.772604 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv" event={"ID":"7dfd62be-e511-4d76-be35-a023febdbda2","Type":"ContainerDied","Data":"d4ea585cdb5e831df09249bae2b834d1cb6265ce26f5f7d2b33d03d2344b4998"} Nov 04 01:27:14.772651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.772628 4312 scope.go:122] "RemoveContainer" containerID="04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa" Nov 04 01:27:14.774228 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.774184 4312 generic.go:356] "Generic (PLEG): container finished" podID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerID="49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e" exitCode=0 Nov 04 01:27:14.774414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.774267 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" Nov 04 01:27:14.774414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.774291 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" event={"ID":"33a9ca41-20ef-4130-bd8c-df377b06c567","Type":"ContainerDied","Data":"49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e"} Nov 04 01:27:14.774414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.774350 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b" event={"ID":"33a9ca41-20ef-4130-bd8c-df377b06c567","Type":"ContainerDied","Data":"069d37feea5113689b0c1b5f9fe31d288a80e0f16610139d326ee97e51c4cb78"} Nov 04 01:27:14.785307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.785235 4312 scope.go:122] "RemoveContainer" containerID="04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa" Nov 04 01:27:14.785604 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:14.785585 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa\": container with ID starting with 04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa not found: ID does not exist" containerID="04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa" Nov 04 01:27:14.785683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.785611 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa"} err="failed to get container status \"04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa\": rpc error: code = NotFound desc = could not find container \"04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa\": container with ID starting with 04804918f2938a348a2c77eac472b24ae1b23a2a0577f8475cf9a354bc16deaa not found: ID does not exist" Nov 04 01:27:14.785683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.785627 4312 scope.go:122] "RemoveContainer" containerID="49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e" Nov 04 01:27:14.795462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.795404 4312 scope.go:122] "RemoveContainer" containerID="49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e" Nov 04 01:27:14.795763 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:14.795736 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e\": container with ID starting with 49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e not found: ID does not exist" containerID="49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e" Nov 04 01:27:14.795895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.795770 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e"} err="failed to get container status \"49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e\": rpc error: code = NotFound desc = could not find container \"49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e\": container with ID starting with 49ebc367166c03d276849c20360e4ad570d70317d826cf38a999b8c50fcf284e not found: ID does not exist" Nov 04 01:27:14.805588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.805514 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv"] Nov 04 01:27:14.814790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.814756 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-qhmvv"] Nov 04 01:27:14.835423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.835393 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b"] Nov 04 01:27:14.839654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:14.839619 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-zfq4b"] Nov 04 01:27:15.499190 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.499149 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh"] Nov 04 01:27:15.499676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.499470 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" podUID="9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" containerName="maas-api" containerID="cri-o://93eba80d40df3ee9bca1adb7bb70154a91a9bd78cbdc56d52fc1ab3a9c8c2418" gracePeriod=30 Nov 04 01:27:15.528786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.528752 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" podUID="9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.126:8443/health\": dial tcp 10.133.0.126:8443: connect: connection refused" Nov 04 01:27:15.537380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.535623 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64"] Nov 04 01:27:15.537380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.536744 4312 cpu_manager.go:414] "containerMap: removing container" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" Nov 04 01:27:15.537380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.536764 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" Nov 04 01:27:15.537380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.536808 4312 cpu_manager.go:414] "containerMap: removing container" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" Nov 04 01:27:15.537380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.536816 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" Nov 04 01:27:15.537380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.537006 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" containerName="istio-proxy" Nov 04 01:27:15.537380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.537027 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" containerName="istio-proxy" Nov 04 01:27:15.542517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.542118 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:15.551163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.551132 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-695ce9\"" Nov 04 01:27:15.558132 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.556464 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64"] Nov 04 01:27:15.560681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.560662 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv"] Nov 04 01:27:15.565825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.565797 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:15.575393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.575371 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-695ce9-dockercfg-cbf9z\"" Nov 04 01:27:15.578403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.578309 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv"] Nov 04 01:27:15.609775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.609746 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:15.609903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.609806 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzm2r\" (UniqueName: \"kubernetes.io/projected/80adeed0-1499-4cfc-a002-114ae3bf45b7-kube-api-access-lzm2r\") pod \"payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:15.609991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.609966 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2gl\" (UniqueName: \"kubernetes.io/projected/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-kube-api-access-kf2gl\") pod \"payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:15.610231 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.610206 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80adeed0-1499-4cfc-a002-114ae3bf45b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:15.722743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.712565 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80adeed0-1499-4cfc-a002-114ae3bf45b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:15.722743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.712688 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:15.722743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.712751 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzm2r\" (UniqueName: \"kubernetes.io/projected/80adeed0-1499-4cfc-a002-114ae3bf45b7-kube-api-access-lzm2r\") pod \"payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:15.722743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.712849 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2gl\" (UniqueName: \"kubernetes.io/projected/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-kube-api-access-kf2gl\") pod \"payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:15.722743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.716171 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80adeed0-1499-4cfc-a002-114ae3bf45b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:15.722743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.716605 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:15.934355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.924651 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2gl\" (UniqueName: \"kubernetes.io/projected/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-kube-api-access-kf2gl\") pod \"payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:15.955359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.943029 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzm2r\" (UniqueName: \"kubernetes.io/projected/80adeed0-1499-4cfc-a002-114ae3bf45b7-kube-api-access-lzm2r\") pod \"payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:15.978357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.962893 4312 generic.go:356] "Generic (PLEG): container finished" podID="9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" containerID="93eba80d40df3ee9bca1adb7bb70154a91a9bd78cbdc56d52fc1ab3a9c8c2418" exitCode=0 Nov 04 01:27:15.978357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:15.963072 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" event={"ID":"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3","Type":"ContainerDied","Data":"93eba80d40df3ee9bca1adb7bb70154a91a9bd78cbdc56d52fc1ab3a9c8c2418"} Nov 04 01:27:16.071278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.071250 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:27:16.116318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.116282 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a9ca41-20ef-4130-bd8c-df377b06c567" path="/var/lib/kubelet/pods/33a9ca41-20ef-4130-bd8c-df377b06c567/volumes" Nov 04 01:27:16.116944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.116912 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfd62be-e511-4d76-be35-a023febdbda2" path="/var/lib/kubelet/pods/7dfd62be-e511-4d76-be35-a023febdbda2/volumes" Nov 04 01:27:16.131679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.131652 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-openshift-service-ca\") pod \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " Nov 04 01:27:16.131805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.131741 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls\") pod \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " Nov 04 01:27:16.131976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.131954 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-openshift-service-ca" pod "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" (UID: "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:16.132020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.131958 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-metrics-certs\") pod \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " Nov 04 01:27:16.132127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.132110 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-kube-api-access-2bv9h\" (UniqueName: \"kubernetes.io/projected/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-kube-api-access-2bv9h\") pod \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\" (UID: \"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3\") " Nov 04 01:27:16.132761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.132732 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:16.133900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.133876 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls" pod "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" (UID: "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:16.134206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.134181 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-kube-api-access-2bv9h" pod "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" (UID: "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3"). InnerVolumeSpecName "kube-api-access-2bv9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:16.134281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.134190 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-metrics-certs" pod "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" (UID: "9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:16.204098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.204022 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:16.211477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.211451 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:16.233767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.233744 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:16.233862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.233769 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bv9h\" (UniqueName: \"kubernetes.io/projected/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-kube-api-access-2bv9h\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:16.233862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.233781 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:16.550766 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:16.550739 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070e48e2_c5d8_4e5a_9c38_ae2ba902b75a.slice/crio-5d52f0ee5359ad20010ec6805cf5ce86d1759340f750fa46f952ac02a4eb9d76 WatchSource:0}: Error finding container 5d52f0ee5359ad20010ec6805cf5ce86d1759340f750fa46f952ac02a4eb9d76: Status 404 returned error can't find the container with id 5d52f0ee5359ad20010ec6805cf5ce86d1759340f750fa46f952ac02a4eb9d76 Nov 04 01:27:16.551658 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.551636 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64"] Nov 04 01:27:16.552018 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:16.551995 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80adeed0_1499_4cfc_a002_114ae3bf45b7.slice/crio-a41269652a967a22e2a7612b0710af1929c716db28cfce624be592b3e8224ab3 WatchSource:0}: Error finding container a41269652a967a22e2a7612b0710af1929c716db28cfce624be592b3e8224ab3: Status 404 returned error can't find the container with id a41269652a967a22e2a7612b0710af1929c716db28cfce624be592b3e8224ab3 Nov 04 01:27:16.560355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.560306 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv"] Nov 04 01:27:16.979538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.979501 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" event={"ID":"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a","Type":"ContainerStarted","Data":"5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2"} Nov 04 01:27:16.979701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.979553 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" event={"ID":"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a","Type":"ContainerStarted","Data":"5d52f0ee5359ad20010ec6805cf5ce86d1759340f750fa46f952ac02a4eb9d76"} Nov 04 01:27:16.979701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.979601 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:16.981262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.981227 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" event={"ID":"80adeed0-1499-4cfc-a002-114ae3bf45b7","Type":"ContainerStarted","Data":"863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd"} Nov 04 01:27:16.981413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.981281 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:16.981413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.981305 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" event={"ID":"80adeed0-1499-4cfc-a002-114ae3bf45b7","Type":"ContainerStarted","Data":"a41269652a967a22e2a7612b0710af1929c716db28cfce624be592b3e8224ab3"} Nov 04 01:27:16.982769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.982749 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" Nov 04 01:27:16.982769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.982758 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh" event={"ID":"9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3","Type":"ContainerDied","Data":"164835ce060dd34aab14b1ffa7c33b551ab69bcc80b2bee465e1a77a4804076a"} Nov 04 01:27:16.982936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:16.982813 4312 scope.go:122] "RemoveContainer" containerID="93eba80d40df3ee9bca1adb7bb70154a91a9bd78cbdc56d52fc1ab3a9c8c2418" Nov 04 01:27:17.001924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.001878 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" podStartSLOduration=2.001860685 podStartE2EDuration="2.001860685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:16.998864714 +0000 UTC m=+1469.021796750" watchObservedRunningTime="2026-11-04 01:27:17.001860685 +0000 UTC m=+1469.024792693" Nov 04 01:27:17.018422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.018394 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh"] Nov 04 01:27:17.024160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.024135 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20a250-67774dcb6d-mrbxh"] Nov 04 01:27:17.037037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.036998 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" podStartSLOduration=2.036985709 podStartE2EDuration="2.036985709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:17.035710582 +0000 UTC m=+1469.058642590" watchObservedRunningTime="2026-11-04 01:27:17.036985709 +0000 UTC m=+1469.059917729" Nov 04 01:27:17.310585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.310550 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752"] Nov 04 01:27:17.310839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.310805 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" podUID="c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" containerName="maas-api" containerID="cri-o://96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e" gracePeriod=30 Nov 04 01:27:17.743484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.743442 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h"] Nov 04 01:27:17.778389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.767672 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w"] Nov 04 01:27:17.797497 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.797465 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:27:17.919144 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.919114 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-kube-api-access-4jfqx\" (UniqueName: \"kubernetes.io/projected/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-kube-api-access-4jfqx\") pod \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " Nov 04 01:27:17.919312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.919180 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls\") pod \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " Nov 04 01:27:17.919312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.919274 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-openshift-service-ca\") pod \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " Nov 04 01:27:17.919312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.919292 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-metrics-certs\") pod \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\" (UID: \"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb\") " Nov 04 01:27:17.919666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.919641 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-openshift-service-ca" pod "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" (UID: "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:17.919949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.919924 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:17.921233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.921210 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-kube-api-access-4jfqx" pod "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" (UID: "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb"). InnerVolumeSpecName "kube-api-access-4jfqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:17.921316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.921245 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls" pod "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" (UID: "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:17.921393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.921357 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-metrics-certs" pod "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" (UID: "c3ae6bd1-8044-4ea8-8748-47afe26bf6eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:17.993286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.993203 4312 generic.go:356] "Generic (PLEG): container finished" podID="c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" containerID="96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e" exitCode=0 Nov 04 01:27:17.993286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.993243 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" event={"ID":"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb","Type":"ContainerDied","Data":"96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e"} Nov 04 01:27:17.993525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.993297 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" event={"ID":"c3ae6bd1-8044-4ea8-8748-47afe26bf6eb","Type":"ContainerDied","Data":"1463735d2790131e7d36ded6616580defde7c34f9494d58b404934a7265c4453"} Nov 04 01:27:17.993525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.993321 4312 scope.go:122] "RemoveContainer" containerID="96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e" Nov 04 01:27:17.993525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:17.993251 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752" Nov 04 01:27:18.005192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.005176 4312 scope.go:122] "RemoveContainer" containerID="96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e" Nov 04 01:27:18.005542 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:18.005519 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e\": container with ID starting with 96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e not found: ID does not exist" containerID="96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e" Nov 04 01:27:18.005628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.005549 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e"} err="failed to get container status \"96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e\": rpc error: code = NotFound desc = could not find container \"96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e\": container with ID starting with 96ed7a5688e97bd62b0f6dcc2fc5944974069cff13873a4eb90710ded1f48b3e not found: ID does not exist" Nov 04 01:27:18.019819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.019796 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752"] Nov 04 01:27:18.020994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.020972 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:18.021092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.021072 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jfqx\" (UniqueName: \"kubernetes.io/projected/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-kube-api-access-4jfqx\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:18.021141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.021092 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:18.025500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.025476 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9902c3-86c6449f59-sv752"] Nov 04 01:27:18.133350 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.133300 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" path="/var/lib/kubelet/pods/9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3/volumes" Nov 04 01:27:18.133866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.133843 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" path="/var/lib/kubelet/pods/c3ae6bd1-8044-4ea8-8748-47afe26bf6eb/volumes" Nov 04 01:27:18.952600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.952568 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm"] Nov 04 01:27:18.953046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.952837 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" containerID="cri-o://1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54" gracePeriod=30 Nov 04 01:27:18.960155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.960131 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7"] Nov 04 01:27:18.960343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.960317 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" containerID="cri-o://6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385" gracePeriod=30 Nov 04 01:27:18.964436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.964411 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857"] Nov 04 01:27:18.964586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:18.964569 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" containerID="cri-o://5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640" gracePeriod=30 Nov 04 01:27:24.262993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.262963 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:27:24.322084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322055 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-data\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322186 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-kube-api-access-gd2gs\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-kube-api-access-gd2gs\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322223 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istiod-ca-cert\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322371 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-data" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.322398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322378 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-socket\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322488 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-podinfo\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322594 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-certs\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322639 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-envoy\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322651 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-socket" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.322770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322742 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-credential-socket\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322824 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322782 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-token\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-token\") pod \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\" (UID: \"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff\") " Nov 04 01:27:24.322880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322835 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-certs" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.323008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.322991 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-credential-socket" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.323140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.323116 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:27:24.323693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.323668 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.323810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.323697 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.323810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.323712 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.323810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.323728 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.324015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.323976 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istiod-ca-cert" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:24.325626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.325597 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-token" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:24.325969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.325929 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-podinfo" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:27:24.326381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.326326 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-kube-api-access-gd2gs" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "kube-api-access-gd2gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:24.328167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.328144 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-envoy" pod "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" (UID: "8a4e203f-a426-4905-a1a2-59c3bf5ce1ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.328699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.328678 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:27:24.425232 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425200 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-token\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.425467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425264 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-credential-socket\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.425467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425372 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-socket\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.425467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425439 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-credential-socket\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425501 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-kube-api-access-jttch\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-kube-api-access-jttch\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425516 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-credential-socket" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425548 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-data\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425591 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-envoy\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425618 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-socket\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425650 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-podinfo\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425655 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-socket" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425684 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istiod-ca-cert\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.425736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425722 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-credential-socket" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425808 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-kube-api-access-6bmwg\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-kube-api-access-6bmwg\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425832 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-data" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425939 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-data\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425975 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-envoy\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.426037 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-certs\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.426151 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istiod-ca-cert\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425938 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-socket" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.425956 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istiod-ca-cert" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.426214 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-podinfo\") pod \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\" (UID: \"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.426293 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-certs\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.426325 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-token\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-token\") pod \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\" (UID: \"e15c9f63-0121-426d-b7ee-7bbae3e8ad82\") " Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.426649 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istiod-ca-cert" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.426765 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-certs" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.427073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427027 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-data" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427229 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-certs" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427325 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427366 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427381 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427396 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427410 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427425 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427439 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427452 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427466 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427480 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427494 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427508 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427523 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2gs\" (UniqueName: \"kubernetes.io/projected/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-kube-api-access-gd2gs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.427929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.427538 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.428722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.428576 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-kube-api-access-6bmwg" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "kube-api-access-6bmwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:24.429102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.429062 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-kube-api-access-jttch" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "kube-api-access-jttch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:24.429283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.429251 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-envoy" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.431086 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.431039 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-podinfo" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:27:24.431255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.431081 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-podinfo" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:27:24.431255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.431097 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-token" pod "e15c9f63-0121-426d-b7ee-7bbae3e8ad82" (UID: "e15c9f63-0121-426d-b7ee-7bbae3e8ad82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:24.431255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.431086 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-envoy" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:24.432462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.432443 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-token" pod "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" (UID: "ac4b7300-8fc6-4173-b3d0-f1a65d296fbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:24.529266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529232 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529264 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529276 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttch\" (UniqueName: \"kubernetes.io/projected/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-kube-api-access-jttch\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529287 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529295 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529305 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmwg\" (UniqueName: \"kubernetes.io/projected/e15c9f63-0121-426d-b7ee-7bbae3e8ad82-kube-api-access-6bmwg\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529314 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529324 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:24.529483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:24.529358 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:25.098036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.097997 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerID="6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385" exitCode=0 Nov 04 01:27:25.098237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.098177 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" event={"ID":"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff","Type":"ContainerDied","Data":"6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385"} Nov 04 01:27:25.098237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.098232 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" event={"ID":"8a4e203f-a426-4905-a1a2-59c3bf5ce1ff","Type":"ContainerDied","Data":"1652f7fc47138a50c3aed4e4cafee542cc9edb97387f4d3698fd6bda88ae9ef6"} Nov 04 01:27:25.098407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.098263 4312 scope.go:122] "RemoveContainer" containerID="6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385" Nov 04 01:27:25.098554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.098534 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7" Nov 04 01:27:25.110188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.109287 4312 generic.go:356] "Generic (PLEG): container finished" podID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerID="5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640" exitCode=0 Nov 04 01:27:25.111494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.110665 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" Nov 04 01:27:25.118175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.118142 4312 generic.go:356] "Generic (PLEG): container finished" podID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerID="1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54" exitCode=0 Nov 04 01:27:25.118305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.118281 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" Nov 04 01:27:25.139385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.129324 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" event={"ID":"e15c9f63-0121-426d-b7ee-7bbae3e8ad82","Type":"ContainerDied","Data":"5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640"} Nov 04 01:27:25.139385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.129391 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857" event={"ID":"e15c9f63-0121-426d-b7ee-7bbae3e8ad82","Type":"ContainerDied","Data":"1ac7ea23f4d183602272c4615f494e1c02da240303c138148237b7991c300d83"} Nov 04 01:27:25.139385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.129416 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" event={"ID":"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd","Type":"ContainerDied","Data":"1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54"} Nov 04 01:27:25.139385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.129440 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm" event={"ID":"ac4b7300-8fc6-4173-b3d0-f1a65d296fbd","Type":"ContainerDied","Data":"26589589d08290743bc6cb6943eb59e3017240334673cafb9744a6e0b781e0aa"} Nov 04 01:27:25.139385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.129995 4312 scope.go:122] "RemoveContainer" containerID="89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98" Nov 04 01:27:25.149815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.149215 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7"] Nov 04 01:27:25.153787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.153755 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-ph2p7"] Nov 04 01:27:25.156318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.156294 4312 scope.go:122] "RemoveContainer" containerID="6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385" Nov 04 01:27:25.156696 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:25.156675 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385\": container with ID starting with 6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385 not found: ID does not exist" containerID="6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385" Nov 04 01:27:25.156771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.156706 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385"} err="failed to get container status \"6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385\": rpc error: code = NotFound desc = could not find container \"6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385\": container with ID starting with 6e3905ef55d40f411c8b51f02474e96579a31eafcbaaaf91017e796397aca385 not found: ID does not exist" Nov 04 01:27:25.156771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.156725 4312 scope.go:122] "RemoveContainer" containerID="89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98" Nov 04 01:27:25.157007 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:25.156989 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98\": container with ID starting with 89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98 not found: ID does not exist" containerID="89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98" Nov 04 01:27:25.157050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.157013 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98"} err="failed to get container status \"89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98\": rpc error: code = NotFound desc = could not find container \"89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98\": container with ID starting with 89ead4199a28ccefe53f00732aebc051f997cacc6fd799dd3c5f7ebdd7f1ef98 not found: ID does not exist" Nov 04 01:27:25.157050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.157028 4312 scope.go:122] "RemoveContainer" containerID="5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640" Nov 04 01:27:25.167285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.167267 4312 scope.go:122] "RemoveContainer" containerID="5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640" Nov 04 01:27:25.167621 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:25.167593 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640\": container with ID starting with 5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640 not found: ID does not exist" containerID="5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640" Nov 04 01:27:25.167690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.167631 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640"} err="failed to get container status \"5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640\": rpc error: code = NotFound desc = could not find container \"5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640\": container with ID starting with 5066971157c1c69139cf65c326451ab1c43a7a169dbddd32959c55ebfd188640 not found: ID does not exist" Nov 04 01:27:25.167690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.167647 4312 scope.go:122] "RemoveContainer" containerID="1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54" Nov 04 01:27:25.169896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.169875 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857"] Nov 04 01:27:25.174054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.174031 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fx857"] Nov 04 01:27:25.178417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.178397 4312 scope.go:122] "RemoveContainer" containerID="1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54" Nov 04 01:27:25.178731 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:25.178705 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54\": container with ID starting with 1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54 not found: ID does not exist" containerID="1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54" Nov 04 01:27:25.178821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.178741 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54"} err="failed to get container status \"1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54\": rpc error: code = NotFound desc = could not find container \"1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54\": container with ID starting with 1ba3e82d0f8f10c508dfa6a1b61ae6d876fae7a2553b37d1ad39997f6bd1ce54 not found: ID does not exist" Nov 04 01:27:25.186175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.186150 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm"] Nov 04 01:27:25.190767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:25.190742 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-sc9qm"] Nov 04 01:27:26.111237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:26.111205 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" path="/var/lib/kubelet/pods/8a4e203f-a426-4905-a1a2-59c3bf5ce1ff/volumes" Nov 04 01:27:26.111837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:26.111781 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" path="/var/lib/kubelet/pods/ac4b7300-8fc6-4173-b3d0-f1a65d296fbd/volumes" Nov 04 01:27:26.112467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:26.112447 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" path="/var/lib/kubelet/pods/e15c9f63-0121-426d-b7ee-7bbae3e8ad82/volumes" Nov 04 01:27:26.113118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:26.113096 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n"] Nov 04 01:27:27.996400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:27.996374 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:27:27.996788 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:27.996443 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:27:29.458888 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:29.458812 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:27:29.967849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.967811 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx"] Nov 04 01:27:29.969791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969760 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" containerName="maas-api" Nov 04 01:27:29.969791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969789 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" containerName="maas-api" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969843 4312 cpu_manager.go:414] "containerMap: removing container" podUID="c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" containerName="maas-api" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969853 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" containerName="maas-api" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969881 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969889 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969914 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969921 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969943 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969951 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.969993 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" Nov 04 01:27:29.970140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.970000 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" Nov 04 01:27:29.972837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.972808 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac6e4c7-aa62-4013-afe7-3e4c0e5e1fe3" containerName="maas-api" Nov 04 01:27:29.972949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.972850 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ae6bd1-8044-4ea8-8748-47afe26bf6eb" containerName="maas-api" Nov 04 01:27:29.972949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.972862 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" Nov 04 01:27:29.972949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.972879 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4b7300-8fc6-4173-b3d0-f1a65d296fbd" containerName="istio-proxy" Nov 04 01:27:29.972949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.972889 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15c9f63-0121-426d-b7ee-7bbae3e8ad82" containerName="istio-proxy" Nov 04 01:27:29.986006 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.985980 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:29.998363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.988263 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx"] Nov 04 01:27:29.999637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:29.999611 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1e0642-openshift-default-dockercfg-jr7kn\"" Nov 04 01:27:30.062917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.062886 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-socket\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063020 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-credential-socket\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063090 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873797b0-f49c-467f-8157-1fccaf30b023-istio-podinfo\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063191 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873797b0-f49c-467f-8157-1fccaf30b023-istiod-ca-cert\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063253 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6v2\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-kube-api-access-rx6v2\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063326 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-certs\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063388 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-istio-token\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063420 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-data\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.063502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.063474 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-envoy\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.209391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.180438 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873797b0-f49c-467f-8157-1fccaf30b023-istiod-ca-cert\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.209676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.183922 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873797b0-f49c-467f-8157-1fccaf30b023-istiod-ca-cert\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.209901 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.209874 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6v2\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-kube-api-access-rx6v2\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.210025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.210011 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-certs\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.210161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.210147 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-istio-token\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.210300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.210285 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-data\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.210461 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-certs\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.221482 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-envoy\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.221551 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-socket\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.221660 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-credential-socket\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.221717 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873797b0-f49c-467f-8157-1fccaf30b023-istio-podinfo\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.222322 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-socket\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.211586 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-data\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.241400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.222609 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-credential-socket\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.264110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.264074 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-istio-token\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.268021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.267985 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873797b0-f49c-467f-8157-1fccaf30b023-istio-podinfo\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.287171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.287101 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6v2\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-kube-api-access-rx6v2\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.310274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.310014 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-envoy\") pod \"e2e-api-b-1e0642-openshift-default-778b777787-hjhrx\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.325668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.325634 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24"] Nov 04 01:27:30.327234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.327207 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4e203f-a426-4905-a1a2-59c3bf5ce1ff" containerName="istio-proxy" Nov 04 01:27:30.338669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.338600 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.381153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.381119 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d0c8e8-openshift-default-dockercfg-2wq84\"" Nov 04 01:27:30.391380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.389840 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24"] Nov 04 01:27:30.431497 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.431457 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-data\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.431686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.431627 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-podinfo\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.431686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.431671 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54w4d\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-kube-api-access-54w4d\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.431819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.431744 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-token\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.431949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.431920 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istiod-ca-cert\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.432008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.431991 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-certs\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.432080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.432037 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-envoy\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.432135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.432105 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-socket\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.432192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.432138 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-credential-socket\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.533592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533559 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istiod-ca-cert\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533622 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-certs\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533671 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-envoy\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533711 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-socket\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533748 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-credential-socket\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533856 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-data\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533923 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-podinfo\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.533954 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54w4d\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-kube-api-access-54w4d\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.534018 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-token\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.534084 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-certs\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.534176 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-credential-socket\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.534193 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-socket\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.534404 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istiod-ca-cert\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.534653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.534495 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-data\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.536461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.536432 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-envoy\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.536570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.536557 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-podinfo\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.551564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.551535 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54w4d\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-kube-api-access-54w4d\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.553089 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.553066 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-token\") pod \"e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:30.613356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.608601 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:30.727642 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:30.726426 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:31.142666 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:31.142640 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873797b0_f49c_467f_8157_1fccaf30b023.slice/crio-990a929ae5e86738d82cf3669fbe46970a2569fdf79c969ad4e641300d9cb3c1 WatchSource:0}: Error finding container 990a929ae5e86738d82cf3669fbe46970a2569fdf79c969ad4e641300d9cb3c1: Status 404 returned error can't find the container with id 990a929ae5e86738d82cf3669fbe46970a2569fdf79c969ad4e641300d9cb3c1 Nov 04 01:27:31.143984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.143959 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx"] Nov 04 01:27:31.145709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.145678 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:31.145776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.145740 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:31.145776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.145769 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:31.160324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.160298 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24"] Nov 04 01:27:31.161463 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:31.161438 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3379949f_d9f1_4c2e_9341_407cbdfc7b85.slice/crio-ad17f318b8394d53e17fb94099f7a42c2ae267ba0f7c05d38bf03f9fe738e796 WatchSource:0}: Error finding container ad17f318b8394d53e17fb94099f7a42c2ae267ba0f7c05d38bf03f9fe738e796: Status 404 returned error can't find the container with id ad17f318b8394d53e17fb94099f7a42c2ae267ba0f7c05d38bf03f9fe738e796 Nov 04 01:27:31.164274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.164235 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:31.164427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.164311 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:31.164427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.164389 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:31.229097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.229059 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" event={"ID":"3379949f-d9f1-4c2e-9341-407cbdfc7b85","Type":"ContainerStarted","Data":"af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa"} Nov 04 01:27:31.229265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.229129 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" event={"ID":"3379949f-d9f1-4c2e-9341-407cbdfc7b85","Type":"ContainerStarted","Data":"ad17f318b8394d53e17fb94099f7a42c2ae267ba0f7c05d38bf03f9fe738e796"} Nov 04 01:27:31.231266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.231239 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" event={"ID":"873797b0-f49c-467f-8157-1fccaf30b023","Type":"ContainerStarted","Data":"d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea"} Nov 04 01:27:31.231405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.231291 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" event={"ID":"873797b0-f49c-467f-8157-1fccaf30b023","Type":"ContainerStarted","Data":"990a929ae5e86738d82cf3669fbe46970a2569fdf79c969ad4e641300d9cb3c1"} Nov 04 01:27:31.293144 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.293089 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podStartSLOduration=1.293075045 podStartE2EDuration="1.293075045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:31.265211351 +0000 UTC m=+1483.288143377" watchObservedRunningTime="2026-11-04 01:27:31.293075045 +0000 UTC m=+1483.316007051" Nov 04 01:27:31.293766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.293726 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podStartSLOduration=2.293714102 podStartE2EDuration="2.293714102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:31.292927314 +0000 UTC m=+1483.315859351" watchObservedRunningTime="2026-11-04 01:27:31.293714102 +0000 UTC m=+1483.316646108" Nov 04 01:27:31.614502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.614459 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:31.727919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:31.727890 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:32.371347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.371301 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h"] Nov 04 01:27:32.376242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.376218 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.390552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.390527 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h"] Nov 04 01:27:32.391080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.391061 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-695ce9-serving-cert\"" Nov 04 01:27:32.523065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.523023 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcsk\" (UniqueName: \"kubernetes.io/projected/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-kube-api-access-mmcsk\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.523237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.523187 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.523237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.523213 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-metrics-certs\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.523318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.523244 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-maas-api-tls\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.619909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.619880 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:32.624807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.624745 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.624807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.624777 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-metrics-certs\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.624807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.624805 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-maas-api-tls\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.624989 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.624962 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcsk\" (UniqueName: \"kubernetes.io/projected/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-kube-api-access-mmcsk\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.625542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.625518 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.627202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.627178 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-metrics-certs\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.627313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.627240 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-maas-api-tls\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.636961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.636936 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcsk\" (UniqueName: \"kubernetes.io/projected/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-kube-api-access-mmcsk\") pod \"maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.697813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.697781 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:32.734765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.734734 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:32.882421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.879849 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" containerID="cri-o://98849059aff2d24761db53d96f0c3bda9ac2abbb8a6767b08d3efd95aa227b15" gracePeriod=45 Nov 04 01:27:32.887629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.886239 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" containerID="cri-o://c8dacd6457ad4901a5131150cdc48b4a17ee0cacb4ef9628892892cee9a9d5ff" gracePeriod=45 Nov 04 01:27:32.967425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:32.925452 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused" Nov 04 01:27:33.125449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.123615 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": dial tcp 10.133.0.78:8000: connect: connection refused" Nov 04 01:27:33.267864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.267831 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h"] Nov 04 01:27:33.313774 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:33.284658 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6f17fc_4a29_4637_92ea_d2a522dee9b3.slice/crio-79eb474692e90118e94d8338bab6628cafa498d6072c1134758726a4caa1577a WatchSource:0}: Error finding container 79eb474692e90118e94d8338bab6628cafa498d6072c1134758726a4caa1577a: Status 404 returned error can't find the container with id 79eb474692e90118e94d8338bab6628cafa498d6072c1134758726a4caa1577a Nov 04 01:27:33.343657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.343623 4312 generic.go:356] "Generic (PLEG): container finished" podID="6a4c5c00-4791-4127-815b-b8715decf72f" containerID="c8dacd6457ad4901a5131150cdc48b4a17ee0cacb4ef9628892892cee9a9d5ff" exitCode=0 Nov 04 01:27:33.343816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.343705 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" event={"ID":"6a4c5c00-4791-4127-815b-b8715decf72f","Type":"ContainerDied","Data":"c8dacd6457ad4901a5131150cdc48b4a17ee0cacb4ef9628892892cee9a9d5ff"} Nov 04 01:27:33.345382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.345312 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" event={"ID":"bb6f17fc-4a29-4637-92ea-d2a522dee9b3","Type":"ContainerStarted","Data":"79eb474692e90118e94d8338bab6628cafa498d6072c1134758726a4caa1577a"} Nov 04 01:27:33.347436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.347411 4312 generic.go:356] "Generic (PLEG): container finished" podID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerID="98849059aff2d24761db53d96f0c3bda9ac2abbb8a6767b08d3efd95aa227b15" exitCode=0 Nov 04 01:27:33.347571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.347538 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" event={"ID":"ad993ef4-97b8-476b-a855-e3be3b893f94","Type":"ContainerDied","Data":"98849059aff2d24761db53d96f0c3bda9ac2abbb8a6767b08d3efd95aa227b15"} Nov 04 01:27:33.347781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.347764 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:33.348138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.348121 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:33.350641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.350621 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:27:33.350758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.350709 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:27:33.362145 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.362128 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:27:33.418175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.418103 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:27:33.496584 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.496553 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-tmp-dir\") pod \"6a4c5c00-4791-4127-815b-b8715decf72f\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " Nov 04 01:27:33.496753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.496649 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-home\") pod \"6a4c5c00-4791-4127-815b-b8715decf72f\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " Nov 04 01:27:33.496753 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.496713 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4c5c00-4791-4127-815b-b8715decf72f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c5c00-4791-4127-815b-b8715decf72f-tls-certs\") pod \"6a4c5c00-4791-4127-815b-b8715decf72f\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " Nov 04 01:27:33.496874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.496781 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-model-cache\") pod \"6a4c5c00-4791-4127-815b-b8715decf72f\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " Nov 04 01:27:33.496947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.496923 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-tmp-dir" pod "6a4c5c00-4791-4127-815b-b8715decf72f" (UID: "6a4c5c00-4791-4127-815b-b8715decf72f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.497027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.496966 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-home" pod "6a4c5c00-4791-4127-815b-b8715decf72f" (UID: "6a4c5c00-4791-4127-815b-b8715decf72f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.497027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.496983 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-dshm\") pod \"6a4c5c00-4791-4127-815b-b8715decf72f\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " Nov 04 01:27:33.497129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.497076 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-model-cache" pod "6a4c5c00-4791-4127-815b-b8715decf72f" (UID: "6a4c5c00-4791-4127-815b-b8715decf72f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.497129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.497081 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4c5c00-4791-4127-815b-b8715decf72f-kube-api-access-qt6lz\" (UniqueName: \"kubernetes.io/projected/6a4c5c00-4791-4127-815b-b8715decf72f-kube-api-access-qt6lz\") pod \"6a4c5c00-4791-4127-815b-b8715decf72f\" (UID: \"6a4c5c00-4791-4127-815b-b8715decf72f\") " Nov 04 01:27:33.498636 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.498595 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.498636 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.498622 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.498840 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.498638 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.499694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.499665 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4c5c00-4791-4127-815b-b8715decf72f-kube-api-access-qt6lz" pod "6a4c5c00-4791-4127-815b-b8715decf72f" (UID: "6a4c5c00-4791-4127-815b-b8715decf72f"). InnerVolumeSpecName "kube-api-access-qt6lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:33.499799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.499737 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-dshm" pod "6a4c5c00-4791-4127-815b-b8715decf72f" (UID: "6a4c5c00-4791-4127-815b-b8715decf72f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.499799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.499782 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4c5c00-4791-4127-815b-b8715decf72f-tls-certs" pod "6a4c5c00-4791-4127-815b-b8715decf72f" (UID: "6a4c5c00-4791-4127-815b-b8715decf72f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:33.600059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600027 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad993ef4-97b8-476b-a855-e3be3b893f94-kube-api-access-gkgc2\" (UniqueName: \"kubernetes.io/projected/ad993ef4-97b8-476b-a855-e3be3b893f94-kube-api-access-gkgc2\") pod \"ad993ef4-97b8-476b-a855-e3be3b893f94\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " Nov 04 01:27:33.600229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600095 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-home\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-home\") pod \"ad993ef4-97b8-476b-a855-e3be3b893f94\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " Nov 04 01:27:33.600229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600213 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-model-cache\") pod \"ad993ef4-97b8-476b-a855-e3be3b893f94\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " Nov 04 01:27:33.600379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600272 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-tmp-dir\") pod \"ad993ef4-97b8-476b-a855-e3be3b893f94\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " Nov 04 01:27:33.600379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600316 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad993ef4-97b8-476b-a855-e3be3b893f94-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad993ef4-97b8-476b-a855-e3be3b893f94-tls-certs\") pod \"ad993ef4-97b8-476b-a855-e3be3b893f94\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " Nov 04 01:27:33.600379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600357 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-home" pod "ad993ef4-97b8-476b-a855-e3be3b893f94" (UID: "ad993ef4-97b8-476b-a855-e3be3b893f94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.600534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600499 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-dshm\") pod \"ad993ef4-97b8-476b-a855-e3be3b893f94\" (UID: \"ad993ef4-97b8-476b-a855-e3be3b893f94\") " Nov 04 01:27:33.600534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600506 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-model-cache" pod "ad993ef4-97b8-476b-a855-e3be3b893f94" (UID: "ad993ef4-97b8-476b-a855-e3be3b893f94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.600534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.600523 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-tmp-dir" pod "ad993ef4-97b8-476b-a855-e3be3b893f94" (UID: "ad993ef4-97b8-476b-a855-e3be3b893f94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.601125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.601101 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt6lz\" (UniqueName: \"kubernetes.io/projected/6a4c5c00-4791-4127-815b-b8715decf72f-kube-api-access-qt6lz\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.601125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.601124 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.601321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.601134 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.601321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.601143 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c5c00-4791-4127-815b-b8715decf72f-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.601321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.601151 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.601321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.601159 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c5c00-4791-4127-815b-b8715decf72f-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.602223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.602200 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad993ef4-97b8-476b-a855-e3be3b893f94-kube-api-access-gkgc2" pod "ad993ef4-97b8-476b-a855-e3be3b893f94" (UID: "ad993ef4-97b8-476b-a855-e3be3b893f94"). InnerVolumeSpecName "kube-api-access-gkgc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:33.602343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.602277 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad993ef4-97b8-476b-a855-e3be3b893f94-tls-certs" pod "ad993ef4-97b8-476b-a855-e3be3b893f94" (UID: "ad993ef4-97b8-476b-a855-e3be3b893f94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:33.602435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.602419 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-dshm" pod "ad993ef4-97b8-476b-a855-e3be3b893f94" (UID: "ad993ef4-97b8-476b-a855-e3be3b893f94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:33.708434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.703184 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgc2\" (UniqueName: \"kubernetes.io/projected/ad993ef4-97b8-476b-a855-e3be3b893f94-kube-api-access-gkgc2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.708434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.703227 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad993ef4-97b8-476b-a855-e3be3b893f94-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:33.708434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:33.703243 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad993ef4-97b8-476b-a855-e3be3b893f94-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:34.357457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.357426 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" Nov 04 01:27:34.357650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.357452 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h" event={"ID":"6a4c5c00-4791-4127-815b-b8715decf72f","Type":"ContainerDied","Data":"d4e416455396bd0794864ddb355e790248e80c460e8b13887d675ff81ec969fb"} Nov 04 01:27:34.357650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.357502 4312 scope.go:122] "RemoveContainer" containerID="c8dacd6457ad4901a5131150cdc48b4a17ee0cacb4ef9628892892cee9a9d5ff" Nov 04 01:27:34.359538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.359503 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" event={"ID":"bb6f17fc-4a29-4637-92ea-d2a522dee9b3","Type":"ContainerStarted","Data":"b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf"} Nov 04 01:27:34.359693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.359571 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:34.361394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.361361 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" Nov 04 01:27:34.361520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.361389 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w" event={"ID":"ad993ef4-97b8-476b-a855-e3be3b893f94","Type":"ContainerDied","Data":"fffee58c28a8842e9a4e1b94ba55579e032cc6df3e709e1efe7be1917cdc6879"} Nov 04 01:27:34.369868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.369851 4312 scope.go:122] "RemoveContainer" containerID="98849059aff2d24761db53d96f0c3bda9ac2abbb8a6767b08d3efd95aa227b15" Nov 04 01:27:34.381435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.381417 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h"] Nov 04 01:27:34.388564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.388543 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/distinct-2-w2-cffa80-kserve-68d59b7548-gqv2h"] Nov 04 01:27:34.414781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.414737 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" podStartSLOduration=1.985544043 podStartE2EDuration="2.414724736s" totalImagesPullingTime="429.180693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:34.410190302 +0000 UTC m=+1486.433122310" watchObservedRunningTime="2026-11-04 01:27:34.414724736 +0000 UTC m=+1486.437656742" Nov 04 01:27:34.430796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.430770 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w"] Nov 04 01:27:34.434516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:34.434494 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/premium-w2-cffa80-kserve-76db488df6-kdw2w"] Nov 04 01:27:35.206370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.206328 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d"] Nov 04 01:27:35.207071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.207056 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" Nov 04 01:27:35.207110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.207072 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" Nov 04 01:27:35.207110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.207104 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" Nov 04 01:27:35.207188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.207110 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" Nov 04 01:27:35.207229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.207219 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" containerName="main" Nov 04 01:27:35.207260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.207237 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" containerName="main" Nov 04 01:27:35.218241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.218211 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.244360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.236826 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1e0642\"" Nov 04 01:27:35.244664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.241686 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1e0642-dockercfg-fchgr\"" Nov 04 01:27:35.253128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.253101 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d"] Nov 04 01:27:35.259439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.259406 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph"] Nov 04 01:27:35.266559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.266532 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph"] Nov 04 01:27:35.266730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.266646 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.272048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.272021 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8wfb\" (UniqueName: \"kubernetes.io/projected/98695cdd-5134-493d-9a8b-c3ac7a826e13-kube-api-access-s8wfb\") pod \"payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.272200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.272183 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98695cdd-5134-493d-9a8b-c3ac7a826e13-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.408355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.401552 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8wfb\" (UniqueName: \"kubernetes.io/projected/98695cdd-5134-493d-9a8b-c3ac7a826e13-kube-api-access-s8wfb\") pod \"payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.408355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.401749 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98695cdd-5134-493d-9a8b-c3ac7a826e13-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.408355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.401788 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.408355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.401831 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmqr\" (UniqueName: \"kubernetes.io/projected/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-kube-api-access-ncmqr\") pod \"payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.435868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.424892 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98695cdd-5134-493d-9a8b-c3ac7a826e13-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.456357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.439036 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8wfb\" (UniqueName: \"kubernetes.io/projected/98695cdd-5134-493d-9a8b-c3ac7a826e13-kube-api-access-s8wfb\") pod \"payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.514466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.514373 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.514466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.514436 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmqr\" (UniqueName: \"kubernetes.io/projected/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-kube-api-access-ncmqr\") pod \"payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.515489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.515458 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.555372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.555284 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmqr\" (UniqueName: \"kubernetes.io/projected/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-kube-api-access-ncmqr\") pod \"payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.576037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.573677 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:35.581282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.581261 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:35.867884 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.867853 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d"] Nov 04 01:27:35.869182 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:35.869161 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98695cdd_5134_493d_9a8b_c3ac7a826e13.slice/crio-eb462968a89ee61a9e962282567616299c675830038d5fbad10bb5a2a51bef27 WatchSource:0}: Error finding container eb462968a89ee61a9e962282567616299c675830038d5fbad10bb5a2a51bef27: Status 404 returned error can't find the container with id eb462968a89ee61a9e962282567616299c675830038d5fbad10bb5a2a51bef27 Nov 04 01:27:35.871551 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:35.871527 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf08c96c_4e00_40b6_8bf0_9d2795eb5fc5.slice/crio-fbf5493de6c1976f36fd85cc386379eea83d8434b79491ff42707a601087bbdd WatchSource:0}: Error finding container fbf5493de6c1976f36fd85cc386379eea83d8434b79491ff42707a601087bbdd: Status 404 returned error can't find the container with id fbf5493de6c1976f36fd85cc386379eea83d8434b79491ff42707a601087bbdd Nov 04 01:27:35.873920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:35.873897 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph"] Nov 04 01:27:36.049210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.049171 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q"] Nov 04 01:27:36.053808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.053776 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.059775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.059746 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d0c8e8\"" Nov 04 01:27:36.061047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.060687 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d0c8e8-dockercfg-d4src\"" Nov 04 01:27:36.065111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.065086 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q"] Nov 04 01:27:36.114556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.114527 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4c5c00-4791-4127-815b-b8715decf72f" path="/var/lib/kubelet/pods/6a4c5c00-4791-4127-815b-b8715decf72f/volumes" Nov 04 01:27:36.115072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.115028 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad993ef4-97b8-476b-a855-e3be3b893f94" path="/var/lib/kubelet/pods/ad993ef4-97b8-476b-a855-e3be3b893f94/volumes" Nov 04 01:27:36.121460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.121436 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw"] Nov 04 01:27:36.125978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.125777 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ec55d-d260-4d44-a5fb-3401330091f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.126544 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.126490 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhnpg\" (UniqueName: \"kubernetes.io/projected/7e1ec55d-d260-4d44-a5fb-3401330091f3-kube-api-access-zhnpg\") pod \"payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.126983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.126963 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:36.132921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.132900 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw"] Nov 04 01:27:36.253369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.245430 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ec55d-d260-4d44-a5fb-3401330091f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.253369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.245571 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhnpg\" (UniqueName: \"kubernetes.io/projected/7e1ec55d-d260-4d44-a5fb-3401330091f3-kube-api-access-zhnpg\") pod \"payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.305361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.303564 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ec55d-d260-4d44-a5fb-3401330091f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.305361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.304562 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhnpg\" (UniqueName: \"kubernetes.io/projected/7e1ec55d-d260-4d44-a5fb-3401330091f3-kube-api-access-zhnpg\") pod \"payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.365878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.365783 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1315cd-6359-47c3-b4c1-7db9e1480078-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:36.366373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.366326 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxcb\" (UniqueName: \"kubernetes.io/projected/ff1315cd-6359-47c3-b4c1-7db9e1480078-kube-api-access-dlxcb\") pod \"payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:36.372636 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.372599 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:36.468119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.467768 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1315cd-6359-47c3-b4c1-7db9e1480078-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:36.468119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.467923 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxcb\" (UniqueName: \"kubernetes.io/projected/ff1315cd-6359-47c3-b4c1-7db9e1480078-kube-api-access-dlxcb\") pod \"payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:36.468700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.468674 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1315cd-6359-47c3-b4c1-7db9e1480078-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:36.473166 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.473119 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" event={"ID":"98695cdd-5134-493d-9a8b-c3ac7a826e13","Type":"ContainerStarted","Data":"84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff"} Nov 04 01:27:36.473276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.473168 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" event={"ID":"98695cdd-5134-493d-9a8b-c3ac7a826e13","Type":"ContainerStarted","Data":"eb462968a89ee61a9e962282567616299c675830038d5fbad10bb5a2a51bef27"} Nov 04 01:27:36.473434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.473391 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:36.475935 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.475908 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" event={"ID":"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5","Type":"ContainerStarted","Data":"647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8"} Nov 04 01:27:36.476046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.475961 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" event={"ID":"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5","Type":"ContainerStarted","Data":"fbf5493de6c1976f36fd85cc386379eea83d8434b79491ff42707a601087bbdd"} Nov 04 01:27:36.476558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.476491 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:36.479986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.479947 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxcb\" (UniqueName: \"kubernetes.io/projected/ff1315cd-6359-47c3-b4c1-7db9e1480078-kube-api-access-dlxcb\") pod \"payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:36.506062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.506006 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" podStartSLOduration=1.505986965 podStartE2EDuration="1.505986965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:36.505407467 +0000 UTC m=+1488.528339475" watchObservedRunningTime="2026-11-04 01:27:36.505986965 +0000 UTC m=+1488.528918972" Nov 04 01:27:36.588091 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.588020 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" podStartSLOduration=1.587998909 podStartE2EDuration="1.587998909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:36.587548894 +0000 UTC m=+1488.610480921" watchObservedRunningTime="2026-11-04 01:27:36.587998909 +0000 UTC m=+1488.610930918" Nov 04 01:27:36.720903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.720851 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q"] Nov 04 01:27:36.721055 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:36.720964 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1ec55d_d260_4d44_a5fb_3401330091f3.slice/crio-00a5de945db5dc5dbd7870b048b50b8fbccf66573377a12263541693930101ab WatchSource:0}: Error finding container 00a5de945db5dc5dbd7870b048b50b8fbccf66573377a12263541693930101ab: Status 404 returned error can't find the container with id 00a5de945db5dc5dbd7870b048b50b8fbccf66573377a12263541693930101ab Nov 04 01:27:36.748200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:36.744849 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:37.096479 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:37.096447 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1315cd_6359_47c3_b4c1_7db9e1480078.slice/crio-90beebb78609a40f9430a4d8ef73af0dab80284d858b1d0a83289aee59775c6f WatchSource:0}: Error finding container 90beebb78609a40f9430a4d8ef73af0dab80284d858b1d0a83289aee59775c6f: Status 404 returned error can't find the container with id 90beebb78609a40f9430a4d8ef73af0dab80284d858b1d0a83289aee59775c6f Nov 04 01:27:37.098038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.098007 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw"] Nov 04 01:27:37.484804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.484711 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" event={"ID":"7e1ec55d-d260-4d44-a5fb-3401330091f3","Type":"ContainerStarted","Data":"959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0"} Nov 04 01:27:37.484804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.484778 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:37.484804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.484797 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" event={"ID":"7e1ec55d-d260-4d44-a5fb-3401330091f3","Type":"ContainerStarted","Data":"00a5de945db5dc5dbd7870b048b50b8fbccf66573377a12263541693930101ab"} Nov 04 01:27:37.486397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.486368 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" event={"ID":"ff1315cd-6359-47c3-b4c1-7db9e1480078","Type":"ContainerStarted","Data":"a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a"} Nov 04 01:27:37.486530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.486412 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" event={"ID":"ff1315cd-6359-47c3-b4c1-7db9e1480078","Type":"ContainerStarted","Data":"90beebb78609a40f9430a4d8ef73af0dab80284d858b1d0a83289aee59775c6f"} Nov 04 01:27:37.486530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.486512 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:37.506569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.506514 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" podStartSLOduration=1.506497867 podStartE2EDuration="1.506497867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:37.501890376 +0000 UTC m=+1489.524822395" watchObservedRunningTime="2026-11-04 01:27:37.506497867 +0000 UTC m=+1489.529429876" Nov 04 01:27:37.520187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:37.520149 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" podStartSLOduration=1.5201378939999999 podStartE2EDuration="1.520137894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:37.519204581 +0000 UTC m=+1489.542136606" watchObservedRunningTime="2026-11-04 01:27:37.520137894 +0000 UTC m=+1489.543069901" Nov 04 01:27:38.444153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:38.444116 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w"] Nov 04 01:27:40.480302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:40.480268 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:27:41.159354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.155260 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" containerID="cri-o://61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c" gracePeriod=45 Nov 04 01:27:41.519498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.519470 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:27:41.534245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.534211 4312 generic.go:356] "Generic (PLEG): container finished" podID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerID="61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c" exitCode=0 Nov 04 01:27:41.534407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.534257 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" Nov 04 01:27:41.534407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.534308 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" event={"ID":"90782a96-bee3-418b-ac21-b65ef47bfa10","Type":"ContainerDied","Data":"61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c"} Nov 04 01:27:41.534407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.534374 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n" event={"ID":"90782a96-bee3-418b-ac21-b65ef47bfa10","Type":"ContainerDied","Data":"d76d4ac72ff743915b1e351526177d45e340955c3729f3284b66c4146e10a987"} Nov 04 01:27:41.534407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.534398 4312 scope.go:122] "RemoveContainer" containerID="61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c" Nov 04 01:27:41.570582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.570548 4312 scope.go:122] "RemoveContainer" containerID="61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c" Nov 04 01:27:41.573750 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:41.573713 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c\": container with ID starting with 61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c not found: ID does not exist" containerID="61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c" Nov 04 01:27:41.573879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.573760 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c"} err="failed to get container status \"61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c\": rpc error: code = NotFound desc = could not find container \"61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c\": container with ID starting with 61c19bf879565d614d7f5eb02cd149e86c64852ca6e57a097718f8e5d98ca75c not found: ID does not exist" Nov 04 01:27:41.634010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.633981 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90782a96-bee3-418b-ac21-b65ef47bfa10-kube-api-access-g8w2k\" (UniqueName: \"kubernetes.io/projected/90782a96-bee3-418b-ac21-b65ef47bfa10-kube-api-access-g8w2k\") pod \"90782a96-bee3-418b-ac21-b65ef47bfa10\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " Nov 04 01:27:41.634165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634021 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-model-cache\") pod \"90782a96-bee3-418b-ac21-b65ef47bfa10\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " Nov 04 01:27:41.634284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634268 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-home\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-home\") pod \"90782a96-bee3-418b-ac21-b65ef47bfa10\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " Nov 04 01:27:41.634474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634288 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-model-cache" pod "90782a96-bee3-418b-ac21-b65ef47bfa10" (UID: "90782a96-bee3-418b-ac21-b65ef47bfa10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:41.634532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634503 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-home" pod "90782a96-bee3-418b-ac21-b65ef47bfa10" (UID: "90782a96-bee3-418b-ac21-b65ef47bfa10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:41.634610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634599 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-tmp-dir\") pod \"90782a96-bee3-418b-ac21-b65ef47bfa10\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " Nov 04 01:27:41.634771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634761 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-dshm\") pod \"90782a96-bee3-418b-ac21-b65ef47bfa10\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " Nov 04 01:27:41.634883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634874 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90782a96-bee3-418b-ac21-b65ef47bfa10-tls-certs\" (UniqueName: \"kubernetes.io/secret/90782a96-bee3-418b-ac21-b65ef47bfa10-tls-certs\") pod \"90782a96-bee3-418b-ac21-b65ef47bfa10\" (UID: \"90782a96-bee3-418b-ac21-b65ef47bfa10\") " Nov 04 01:27:41.635115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.634782 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-tmp-dir" pod "90782a96-bee3-418b-ac21-b65ef47bfa10" (UID: "90782a96-bee3-418b-ac21-b65ef47bfa10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:41.635586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.635568 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:41.635686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.635677 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:41.635739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.635732 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:41.636260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.636235 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90782a96-bee3-418b-ac21-b65ef47bfa10-kube-api-access-g8w2k" pod "90782a96-bee3-418b-ac21-b65ef47bfa10" (UID: "90782a96-bee3-418b-ac21-b65ef47bfa10"). InnerVolumeSpecName "kube-api-access-g8w2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:41.636849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.636771 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-dshm" pod "90782a96-bee3-418b-ac21-b65ef47bfa10" (UID: "90782a96-bee3-418b-ac21-b65ef47bfa10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:41.636939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.636825 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90782a96-bee3-418b-ac21-b65ef47bfa10-tls-certs" pod "90782a96-bee3-418b-ac21-b65ef47bfa10" (UID: "90782a96-bee3-418b-ac21-b65ef47bfa10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:41.737327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.737289 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8w2k\" (UniqueName: \"kubernetes.io/projected/90782a96-bee3-418b-ac21-b65ef47bfa10-kube-api-access-g8w2k\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:41.737327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.737321 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90782a96-bee3-418b-ac21-b65ef47bfa10-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:41.737327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.737344 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90782a96-bee3-418b-ac21-b65ef47bfa10-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:41.927716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.927675 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n"] Nov 04 01:27:41.931044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:41.931012 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-875528/premium-w5-875528-kserve-6f476df68b-jpt2n"] Nov 04 01:27:42.112106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:42.112070 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" path="/var/lib/kubelet/pods/90782a96-bee3-418b-ac21-b65ef47bfa10/volumes" Nov 04 01:27:43.444954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.444915 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" podUID="4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" containerName="payload-pre-processing" containerID="cri-o://2e1dea7aadb697ba14e1cfcce8dcf2b95f45d53e2f9f6855726b1f24b9e3f95a" gracePeriod=25 Nov 04 01:27:43.580204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.580174 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp"] Nov 04 01:27:43.586160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.586132 4312 generic.go:356] "Generic (PLEG): container finished" podID="4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" containerID="2e1dea7aadb697ba14e1cfcce8dcf2b95f45d53e2f9f6855726b1f24b9e3f95a" exitCode=0 Nov 04 01:27:43.586274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.586169 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" event={"ID":"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898","Type":"ContainerDied","Data":"2e1dea7aadb697ba14e1cfcce8dcf2b95f45d53e2f9f6855726b1f24b9e3f95a"} Nov 04 01:27:43.825269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.825248 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:27:43.925360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.921698 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-kube-api-access-948j7\" (UniqueName: \"kubernetes.io/projected/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-kube-api-access-948j7\") pod \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " Nov 04 01:27:43.925360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.921784 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-plugins-config-volume\") pod \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\" (UID: \"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898\") " Nov 04 01:27:43.954384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:43.950543 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-plugins-config-volume" pod "4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" (UID: "4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:44.012358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.004571 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-kube-api-access-948j7" pod "4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" (UID: "4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898"). InnerVolumeSpecName "kube-api-access-948j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:44.029390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.024125 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-948j7\" (UniqueName: \"kubernetes.io/projected/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-kube-api-access-948j7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:44.029390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.024164 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:44.595872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.595842 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" Nov 04 01:27:44.595872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.595864 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w" event={"ID":"4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898","Type":"ContainerDied","Data":"460e1cf6dd0d3e9c129b699e550a9b1f2bda99b900005e98891de5ebd53b4bdc"} Nov 04 01:27:44.596371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.595930 4312 scope.go:122] "RemoveContainer" containerID="2e1dea7aadb697ba14e1cfcce8dcf2b95f45d53e2f9f6855726b1f24b9e3f95a" Nov 04 01:27:44.616873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.616846 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w"] Nov 04 01:27:44.623945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:44.623923 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-029e8c-8d6b95699-nb99w"] Nov 04 01:27:46.033052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.033020 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z"] Nov 04 01:27:46.034572 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.034547 4312 cpu_manager.go:414] "containerMap: removing container" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" Nov 04 01:27:46.034724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.034702 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" Nov 04 01:27:46.034864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.034740 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" containerName="payload-pre-processing" Nov 04 01:27:46.034864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.034750 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" containerName="payload-pre-processing" Nov 04 01:27:46.034976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.034898 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" containerName="payload-pre-processing" Nov 04 01:27:46.034976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.034924 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="90782a96-bee3-418b-ac21-b65ef47bfa10" containerName="main" Nov 04 01:27:46.041531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.041353 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.056509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.056468 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1e0642-serving-cert\"" Nov 04 01:27:46.069046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.069020 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z"] Nov 04 01:27:46.115691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.115657 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898" path="/var/lib/kubelet/pods/4a9ed4bb-c6c5-4a50-8283-1cab9b3f1898/volumes" Nov 04 01:27:46.158985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.158952 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-maas-api-tls\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.159220 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.159197 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cqsb\" (UniqueName: \"kubernetes.io/projected/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-kube-api-access-9cqsb\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.159391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.159319 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-metrics-certs\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.159641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.159613 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.261415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.260942 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqsb\" (UniqueName: \"kubernetes.io/projected/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-kube-api-access-9cqsb\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.261415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.261026 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-metrics-certs\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.261415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.261178 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.261415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.261283 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-maas-api-tls\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.268365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.266154 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.268365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.266911 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-maas-api-tls\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.289397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.269419 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-metrics-certs\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.296169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.296115 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqsb\" (UniqueName: \"kubernetes.io/projected/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-kube-api-access-9cqsb\") pod \"maas-api-e2e-api-b-1e0642-7f66948678-98t5z\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.376808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.376768 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:46.687581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.687499 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z"] Nov 04 01:27:46.715011 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:27:46.714974 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6e6b00_37a7_41f6_8b73_0da04ecb6b41.slice/crio-86844ca082410ddc75536440e699fd6ef2acb21eac6b4fe0fcefc85b4d85f99b WatchSource:0}: Error finding container 86844ca082410ddc75536440e699fd6ef2acb21eac6b4fe0fcefc85b4d85f99b: Status 404 returned error can't find the container with id 86844ca082410ddc75536440e699fd6ef2acb21eac6b4fe0fcefc85b4d85f99b Nov 04 01:27:46.828207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.824710 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl"] Nov 04 01:27:46.831680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.831656 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:46.839195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.836881 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d0c8e8-serving-cert\"" Nov 04 01:27:46.840879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.840447 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl"] Nov 04 01:27:46.919773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.919740 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:46.920035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.920011 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b43751-6a7d-492a-9b6c-5d631d97abe6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:46.920148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.920126 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2q8x\" (UniqueName: \"kubernetes.io/projected/e9b43751-6a7d-492a-9b6c-5d631d97abe6-kube-api-access-f2q8x\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:46.920293 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:46.920271 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-metrics-certs\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.072541 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.072786 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b43751-6a7d-492a-9b6c-5d631d97abe6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.072871 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2q8x\" (UniqueName: \"kubernetes.io/projected/e9b43751-6a7d-492a-9b6c-5d631d97abe6-kube-api-access-f2q8x\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.082987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.073012 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-metrics-certs\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.082987 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:47.081623 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-serving-cert: secret "maas-api-e2e-api-b-d0c8e8-serving-cert" not found Nov 04 01:27:47.082987 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:47.081718 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls podName:e9b43751-6a7d-492a-9b6c-5d631d97abe6 nodeName:}" failed. No retries permitted until 2026-11-04 01:27:47.581679975 +0000 UTC m=+1499.604611984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls") pod "maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" (UID: "e9b43751-6a7d-492a-9b6c-5d631d97abe6") : secret "maas-api-e2e-api-b-d0c8e8-serving-cert" not found Nov 04 01:27:47.084518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.083847 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b43751-6a7d-492a-9b6c-5d631d97abe6-openshift-service-ca\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.110975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.110943 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-metrics-certs\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.131914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.129514 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2q8x\" (UniqueName: \"kubernetes.io/projected/e9b43751-6a7d-492a-9b6c-5d631d97abe6-kube-api-access-f2q8x\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.576357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.576153 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:27:47.621162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.618585 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.627814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.627783 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:27:47.655360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.649995 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls\") pod \"maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:47.684402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.683486 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" event={"ID":"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41","Type":"ContainerStarted","Data":"11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41"} Nov 04 01:27:47.684402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.683616 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" event={"ID":"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41","Type":"ContainerStarted","Data":"86844ca082410ddc75536440e699fd6ef2acb21eac6b4fe0fcefc85b4d85f99b"} Nov 04 01:27:47.684402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.683872 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:47.708924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.706651 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" podStartSLOduration=1.152270282 podStartE2EDuration="1.706637694s" totalImagesPullingTime="554.367412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:47.702811153 +0000 UTC m=+1499.725743171" watchObservedRunningTime="2026-11-04 01:27:47.706637694 +0000 UTC m=+1499.729569700" Nov 04 01:27:47.755131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:47.755106 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:48.196893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:48.196818 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl"] Nov 04 01:27:48.500954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:48.500878 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:27:48.501455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:48.501436 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:27:48.592358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:48.581527 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" podUID="08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" containerName="payload-processing" containerID="cri-o://95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" gracePeriod=25 Nov 04 01:27:48.798627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:48.798574 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" event={"ID":"e9b43751-6a7d-492a-9b6c-5d631d97abe6","Type":"ContainerStarted","Data":"3191c5829fcb98d1b5fc4a47061516ee9075559af1ce3410a007aa5354916ad8"} Nov 04 01:27:49.089967 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.089943 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:27:49.095373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.095320 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-kube-api-access-rh5lp\" (UniqueName: \"kubernetes.io/projected/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-kube-api-access-rh5lp\") pod \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " Nov 04 01:27:49.095668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.095638 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-plugins-config-volume\") pod \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\" (UID: \"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce\") " Nov 04 01:27:49.097069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.096539 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-plugins-config-volume" pod "08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" (UID: "08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:49.105437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.105406 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-kube-api-access-rh5lp" pod "08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" (UID: "08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce"). InnerVolumeSpecName "kube-api-access-rh5lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:49.197711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.197667 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh5lp\" (UniqueName: \"kubernetes.io/projected/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-kube-api-access-rh5lp\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:49.197711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.197712 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:49.808790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.808760 4312 scope.go:122] "RemoveContainer" containerID="95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" Nov 04 01:27:49.813231 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.813204 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" event={"ID":"e9b43751-6a7d-492a-9b6c-5d631d97abe6","Type":"ContainerStarted","Data":"afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411"} Nov 04 01:27:49.813394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.813366 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:49.814868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.814844 4312 generic.go:356] "Generic (PLEG): container finished" podID="08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" containerID="95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" exitCode=0 Nov 04 01:27:49.814968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.814887 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" Nov 04 01:27:49.814968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.814922 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" event={"ID":"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce","Type":"ContainerDied","Data":"95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f"} Nov 04 01:27:49.815056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.814967 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp" event={"ID":"08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce","Type":"ContainerDied","Data":"f89b7a56ca42c9bba742e7782cd8b976be46afa9292789cf0ffc40b65d4e1d9a"} Nov 04 01:27:49.815056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.814994 4312 scope.go:122] "RemoveContainer" containerID="95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" Nov 04 01:27:49.820586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.820303 4312 scope.go:122] "RemoveContainer" containerID="95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" Nov 04 01:27:49.820649 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:49.820589 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f\": container with ID starting with 95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f not found: ID does not exist" containerID="95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" Nov 04 01:27:49.820649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.820618 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f"} err="failed to get container status \"95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f\": rpc error: code = NotFound desc = could not find container \"95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f\": container with ID starting with 95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f not found: ID does not exist" Nov 04 01:27:49.820736 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:49.820673 4312 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp_openshift-ingress_08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce_0 in pod sandbox f89b7a56ca42c9bba742e7782cd8b976be46afa9292789cf0ffc40b65d4e1d9a from index: no such id: '95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f'" containerID="95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" Nov 04 01:27:49.820736 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:49.820703 4312 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp_openshift-ingress_08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce_0 in pod sandbox f89b7a56ca42c9bba742e7782cd8b976be46afa9292789cf0ffc40b65d4e1d9a from index: no such id: '95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f'" containerID="95fdb8e62854fe5f24f568018251a16a905fb9fdb9efe1dd499c6719d374319f" Nov 04 01:27:49.837182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.837137 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" podStartSLOduration=3.232676508 podStartE2EDuration="3.837122276s" totalImagesPullingTime="604.445768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:49.832582157 +0000 UTC m=+1501.855514199" watchObservedRunningTime="2026-11-04 01:27:49.837122276 +0000 UTC m=+1501.860054283" Nov 04 01:27:49.855135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.855106 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": read tcp 10.133.0.2:40826->10.133.0.98:15021: read: connection reset by peer" start-of-body= Nov 04 01:27:49.855327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.855296 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": read tcp 10.133.0.2:40826->10.133.0.98:15021: read: connection reset by peer" Nov 04 01:27:49.857801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.857772 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp"] Nov 04 01:27:49.867775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:49.867750 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-875528-b6f677994-t4bsp"] Nov 04 01:27:50.113078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:50.113006 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" path="/var/lib/kubelet/pods/08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce/volumes" Nov 04 01:27:50.894059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:50.894029 4312 generic.go:356] "Generic (PLEG): container finished" podID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerID="f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc" exitCode=0 Nov 04 01:27:50.894583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:50.894096 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" event={"ID":"67f775a2-6d6c-422c-be49-5eca668d52c8","Type":"ContainerDied","Data":"f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc"} Nov 04 01:27:50.894675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:50.894658 4312 scope.go:122] "RemoveContainer" containerID="f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc" Nov 04 01:27:50.896352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:50.896312 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:50.896429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:50.896394 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:50.896486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:50.896430 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:51.903228 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:51.903198 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" event={"ID":"67f775a2-6d6c-422c-be49-5eca668d52c8","Type":"ContainerStarted","Data":"f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f"} Nov 04 01:27:52.295526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:52.295498 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:27:52.295719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:52.295696 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:27:52.295784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:52.295762 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 04 01:27:53.296779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:53.296740 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:27:53.297644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:53.297605 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 04 01:27:53.643182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:53.643068 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6"] Nov 04 01:27:53.940573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:53.940473 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:27:54.355508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:54.355472 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 04 01:27:54.356149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:54.356115 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 04 01:27:54.373362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:54.359098 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" containerID="cri-o://f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f" gracePeriod=30 Nov 04 01:27:55.920525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:55.920496 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:27:56.025355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:56.021576 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:56.331364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:56.304224 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:57.028359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:57.027358 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:27:57.028359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:57.028093 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:27:57.405413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:57.404535 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:57.413366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:57.411093 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:27:57.429505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:57.425406 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:27:57.529472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:57.528462 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:58.227575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.225200 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:27:58.649202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.649162 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 04 01:27:58.661773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.661740 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" podUID="f659cf1c-b69f-436b-8bf0-9bbddd61383d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Nov 04 01:27:58.680950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.661694 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:58.691425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.689934 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:27:58.784422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.784005 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27"] Nov 04 01:27:58.813359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.785562 4312 cpu_manager.go:414] "containerMap: removing container" podUID="08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" containerName="payload-processing" Nov 04 01:27:58.813359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.785611 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" containerName="payload-processing" Nov 04 01:27:58.813359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.788690 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="08660fcb-c4ba-4d07-b709-2eb9d6fcf4ce" containerName="payload-processing" Nov 04 01:27:58.864358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.863484 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" podUID="98695cdd-5134-493d-9a8b-c3ac7a826e13" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 04 01:27:58.902414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.901723 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:27:58.902414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.901765 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27"] Nov 04 01:27:58.921388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.902777 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:58.921388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.914914 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6ecd1f-dockercfg-xgrsq\"" Nov 04 01:27:58.932667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.930584 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn"] Nov 04 01:27:58.939012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.937310 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:58.948357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.943881 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6ecd1f\"" Nov 04 01:27:58.966167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.966139 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn"] Nov 04 01:27:58.966461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.966443 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:27:58.969315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.969289 4312 generic.go:356] "Generic (PLEG): container finished" podID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerID="b8c4188caf23f249b19a0c5394f02a7e41dc14b43c429f24d39eb8475ea23a89" exitCode=2 Nov 04 01:27:58.970674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.970648 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" event={"ID":"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc","Type":"ContainerDied","Data":"b8c4188caf23f249b19a0c5394f02a7e41dc14b43c429f24d39eb8475ea23a89"} Nov 04 01:27:58.971188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.971169 4312 scope.go:122] "RemoveContainer" containerID="b8c4188caf23f249b19a0c5394f02a7e41dc14b43c429f24d39eb8475ea23a89" Nov 04 01:27:58.982502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.982468 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:58.982704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.982683 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:27:58.983908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:58.983890 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:27:59.125578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.125519 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8gc\" (UniqueName: \"kubernetes.io/projected/6c5e7b78-7288-466e-b786-4a54459e6fef-kube-api-access-4x8gc\") pod \"payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:59.126938 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.126909 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5e7b78-7288-466e-b786-4a54459e6fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:59.130162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.130093 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz" Nov 04 01:27:59.231195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.231064 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb64r\" (UniqueName: \"kubernetes.io/projected/aa9f1ed9-7e45-4918-ac73-075ef310e33d-kube-api-access-mb64r\") pod \"payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:59.231768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.231743 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5e7b78-7288-466e-b786-4a54459e6fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:59.233914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.233888 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1ed9-7e45-4918-ac73-075ef310e33d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:59.234344 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.234310 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8gc\" (UniqueName: \"kubernetes.io/projected/6c5e7b78-7288-466e-b786-4a54459e6fef-kube-api-access-4x8gc\") pod \"payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:59.235284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.235240 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5e7b78-7288-466e-b786-4a54459e6fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:59.328852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.328788 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8gc\" (UniqueName: \"kubernetes.io/projected/6c5e7b78-7288-466e-b786-4a54459e6fef-kube-api-access-4x8gc\") pod \"payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:59.336459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.336418 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb64r\" (UniqueName: \"kubernetes.io/projected/aa9f1ed9-7e45-4918-ac73-075ef310e33d-kube-api-access-mb64r\") pod \"payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:59.337000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.336969 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1ed9-7e45-4918-ac73-075ef310e33d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:59.338197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.338171 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1ed9-7e45-4918-ac73-075ef310e33d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:59.417712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.417682 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb64r\" (UniqueName: \"kubernetes.io/projected/aa9f1ed9-7e45-4918-ac73-075ef310e33d-kube-api-access-mb64r\") pod \"payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:59.624365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.594101 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:27:59.624365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.602803 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:27:59.626369 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:27:59.625387 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:27:59.630157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:27:59.629319 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:28:00.242356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:00.239216 4312 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-42sbp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:00.242356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:00.239263 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:00.246255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:00.246235 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27"] Nov 04 01:28:00.246423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:00.246404 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" event={"ID":"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc","Type":"ContainerStarted","Data":"d058c2b3ee026a158bc5e83f2dab3d9913176b5e1047d293ae6cebae13faa638"} Nov 04 01:28:00.246513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:00.246502 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:28:00.246646 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:00.246635 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn"] Nov 04 01:28:01.239302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.239130 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" event={"ID":"6c5e7b78-7288-466e-b786-4a54459e6fef","Type":"ContainerStarted","Data":"1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209"} Nov 04 01:28:01.239302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.239195 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" event={"ID":"6c5e7b78-7288-466e-b786-4a54459e6fef","Type":"ContainerStarted","Data":"e629ca7461e26aaea9c4899096874ea779bc71a9723627b554364911016d012e"} Nov 04 01:28:01.239302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.239224 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:28:01.241048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.241013 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" event={"ID":"aa9f1ed9-7e45-4918-ac73-075ef310e33d","Type":"ContainerStarted","Data":"e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704"} Nov 04 01:28:01.241186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.241074 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" event={"ID":"aa9f1ed9-7e45-4918-ac73-075ef310e33d","Type":"ContainerStarted","Data":"2ae976862a685b9eaaaeeb8e12cefc3ae82fcda87c99b0158f35174d86639e33"} Nov 04 01:28:01.241303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.241275 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:28:01.241926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.241907 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:28:01.261399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.261310 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" podStartSLOduration=4.261291278 podStartE2EDuration="4.261291278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:01.257594732 +0000 UTC m=+1513.280526824" watchObservedRunningTime="2026-11-04 01:28:01.261291278 +0000 UTC m=+1513.284223287" Nov 04 01:28:01.281251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.281197 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" podStartSLOduration=4.2811815939999995 podStartE2EDuration="4.281181594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:01.279051969 +0000 UTC m=+1513.301984200" watchObservedRunningTime="2026-11-04 01:28:01.281181594 +0000 UTC m=+1513.304113601" Nov 04 01:28:01.359292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:01.359244 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2"] Nov 04 01:28:04.675915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.675883 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:28:04.779768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.779728 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-token\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-token\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.779942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.779807 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-credential-socket\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.779942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.779848 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-podinfo\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.779942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.779899 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f775a2-6d6c-422c-be49-5eca668d52c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f775a2-6d6c-422c-be49-5eca668d52c8-istiod-ca-cert\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.780123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780005 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-certs\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.780188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780134 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-credential-socket" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:04.780244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780200 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-data\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.780348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780316 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-kube-api-access-6th9n\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-kube-api-access-6th9n\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.780414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780365 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-envoy\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.780414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780318 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-certs" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:04.780529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780389 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f775a2-6d6c-422c-be49-5eca668d52c8-istiod-ca-cert" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:04.780529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780419 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-socket\") pod \"67f775a2-6d6c-422c-be49-5eca668d52c8\" (UID: \"67f775a2-6d6c-422c-be49-5eca668d52c8\") " Nov 04 01:28:04.780638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780580 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-data" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:04.780697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.780657 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-socket" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:04.781419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.781390 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.781419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.781419 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.781587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.781433 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f775a2-6d6c-422c-be49-5eca668d52c8-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.781587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.781450 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.781587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.781465 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.782467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.782390 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-token" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:04.782882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.782863 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-kube-api-access-6th9n" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "kube-api-access-6th9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:04.782980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.782965 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-podinfo" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:04.783047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.783035 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-envoy" pod "67f775a2-6d6c-422c-be49-5eca668d52c8" (UID: "67f775a2-6d6c-422c-be49-5eca668d52c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:04.886549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.885200 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6th9n\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-kube-api-access-6th9n\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.886549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.885241 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.886549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.885257 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.886549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:04.885271 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f775a2-6d6c-422c-be49-5eca668d52c8-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.275509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.275473 4312 generic.go:356] "Generic (PLEG): container finished" podID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerID="f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f" exitCode=0 Nov 04 01:28:05.275752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.275543 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" Nov 04 01:28:05.275752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.275551 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" event={"ID":"67f775a2-6d6c-422c-be49-5eca668d52c8","Type":"ContainerDied","Data":"f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f"} Nov 04 01:28:05.275752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.275608 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6" event={"ID":"67f775a2-6d6c-422c-be49-5eca668d52c8","Type":"ContainerDied","Data":"0eea3e187618b74df9577ea789c3bb6526cf3c0b299ccb5ef2b47381ab1ebe0f"} Nov 04 01:28:05.275752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.275630 4312 scope.go:122] "RemoveContainer" containerID="f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f" Nov 04 01:28:05.277501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.277475 4312 generic.go:356] "Generic (PLEG): container finished" podID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerID="86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b" exitCode=0 Nov 04 01:28:05.277612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.277546 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" event={"ID":"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910","Type":"ContainerDied","Data":"86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b"} Nov 04 01:28:05.277975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.277958 4312 scope.go:122] "RemoveContainer" containerID="86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b" Nov 04 01:28:05.280178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.280144 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:05.280300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.280236 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:05.280300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.280285 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:05.291758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.291727 4312 scope.go:122] "RemoveContainer" containerID="f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc" Nov 04 01:28:05.302193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.302171 4312 scope.go:122] "RemoveContainer" containerID="f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f" Nov 04 01:28:05.302526 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:05.302494 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f\": container with ID starting with f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f not found: ID does not exist" containerID="f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f" Nov 04 01:28:05.302628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.302537 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f"} err="failed to get container status \"f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f\": rpc error: code = NotFound desc = could not find container \"f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f\": container with ID starting with f3a003808fba815467d64e3e85e1e01ba2086705b3dd8f5bf845b01d08af1f9f not found: ID does not exist" Nov 04 01:28:05.302628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.302562 4312 scope.go:122] "RemoveContainer" containerID="f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc" Nov 04 01:28:05.302839 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:05.302816 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc\": container with ID starting with f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc not found: ID does not exist" containerID="f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc" Nov 04 01:28:05.302910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.302846 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc"} err="failed to get container status \"f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc\": rpc error: code = NotFound desc = could not find container \"f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc\": container with ID starting with f051a8ff4c4e376987f6cc91905f4d1e8feddaeeded700a2c80567a350fb96fc not found: ID does not exist" Nov 04 01:28:05.337490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.337448 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6"] Nov 04 01:28:05.343470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:05.343443 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-029e8c-openshift-default-6c7c5d9f89-sfqq6"] Nov 04 01:28:06.112450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.112416 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" path="/var/lib/kubelet/pods/67f775a2-6d6c-422c-be49-5eca668d52c8/volumes" Nov 04 01:28:06.364460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.361417 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" podUID="3b40aff9-f6df-482d-b40d-ec023527eb7b" containerName="payload-processing" containerID="cri-o://3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde" gracePeriod=25 Nov 04 01:28:06.475262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.464399 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" event={"ID":"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910","Type":"ContainerStarted","Data":"055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e"} Nov 04 01:28:06.508394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.504318 4312 generic.go:356] "Generic (PLEG): container finished" podID="abc63742-f448-4195-bbe2-d15647027916" containerID="fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1" exitCode=0 Nov 04 01:28:06.508394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.504396 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerDied","Data":"fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1"} Nov 04 01:28:06.508394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.504465 4312 scope.go:122] "RemoveContainer" containerID="130915f29c020dc2ed76bf5521f8e38f88aa2c989e86586cfba75df8d64347cb" Nov 04 01:28:06.513175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.510076 4312 scope.go:122] "RemoveContainer" containerID="fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1" Nov 04 01:28:06.513796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.513741 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:06.513920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.513849 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:06.514059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.513963 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:06.788236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.783709 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:28:06.788236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.783829 4312 patch_prober.go:29] interesting pod/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 01:28:06.788236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.783875 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 01:28:06.885352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:06.884938 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:28:07.002668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.002635 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b40aff9-f6df-482d-b40d-ec023527eb7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b40aff9-f6df-482d-b40d-ec023527eb7b-plugins-config-volume\") pod \"3b40aff9-f6df-482d-b40d-ec023527eb7b\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " Nov 04 01:28:07.002872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.002723 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b40aff9-f6df-482d-b40d-ec023527eb7b-kube-api-access-bjpr7\" (UniqueName: \"kubernetes.io/projected/3b40aff9-f6df-482d-b40d-ec023527eb7b-kube-api-access-bjpr7\") pod \"3b40aff9-f6df-482d-b40d-ec023527eb7b\" (UID: \"3b40aff9-f6df-482d-b40d-ec023527eb7b\") " Nov 04 01:28:07.003085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.003050 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b40aff9-f6df-482d-b40d-ec023527eb7b-plugins-config-volume" pod "3b40aff9-f6df-482d-b40d-ec023527eb7b" (UID: "3b40aff9-f6df-482d-b40d-ec023527eb7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:07.003461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.003440 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b40aff9-f6df-482d-b40d-ec023527eb7b-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:07.005500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.005478 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b40aff9-f6df-482d-b40d-ec023527eb7b-kube-api-access-bjpr7" pod "3b40aff9-f6df-482d-b40d-ec023527eb7b" (UID: "3b40aff9-f6df-482d-b40d-ec023527eb7b"). InnerVolumeSpecName "kube-api-access-bjpr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:07.104701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.104659 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjpr7\" (UniqueName: \"kubernetes.io/projected/3b40aff9-f6df-482d-b40d-ec023527eb7b-kube-api-access-bjpr7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:07.531640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.531521 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerStarted","Data":"5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f"} Nov 04 01:28:07.534850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.534822 4312 generic.go:356] "Generic (PLEG): container finished" podID="3b40aff9-f6df-482d-b40d-ec023527eb7b" containerID="3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde" exitCode=0 Nov 04 01:28:07.535000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.534951 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" Nov 04 01:28:07.535000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.534975 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" event={"ID":"3b40aff9-f6df-482d-b40d-ec023527eb7b","Type":"ContainerDied","Data":"3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde"} Nov 04 01:28:07.535127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.535048 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2" event={"ID":"3b40aff9-f6df-482d-b40d-ec023527eb7b","Type":"ContainerDied","Data":"4437dd1430bc97fd3eaee774f9ca66c8b2d2547d9c1a12affc0b86daac19a7d3"} Nov 04 01:28:07.535127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.535083 4312 scope.go:122] "RemoveContainer" containerID="3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde" Nov 04 01:28:07.555723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.555696 4312 scope.go:122] "RemoveContainer" containerID="3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde" Nov 04 01:28:07.559575 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:07.559543 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde\": container with ID starting with 3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde not found: ID does not exist" containerID="3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde" Nov 04 01:28:07.559783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.559754 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde"} err="failed to get container status \"3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde\": rpc error: code = NotFound desc = could not find container \"3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde\": container with ID starting with 3ce333523526a2096c017dd7f4d5d8289aaabbaf84382bc0a4a79b3bb6ee2fde not found: ID does not exist" Nov 04 01:28:07.595682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.595579 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2"] Nov 04 01:28:07.599997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.599955 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-3bf4f9-8554dbfbc8-fstd2"] Nov 04 01:28:07.793429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.792554 4312 patch_prober.go:29] interesting pod/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 01:28:07.793429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:07.792618 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 01:28:08.131882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.131531 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:28:08.131882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.131592 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:28:08.133962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.133934 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b40aff9-f6df-482d-b40d-ec023527eb7b" path="/var/lib/kubelet/pods/3b40aff9-f6df-482d-b40d-ec023527eb7b/volumes" Nov 04 01:28:08.134453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.134436 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:28:08.135953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.135926 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:28:08.316485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.315965 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq"] Nov 04 01:28:08.316485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.316204 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" podUID="15f1368b-899a-4ca7-9edd-57b77f4657b5" containerName="maas-api" containerID="cri-o://5157ddb64065f50d2310ab044458bff8ab5da696b1854f23880f5eba2f2508b0" gracePeriod=30 Nov 04 01:28:08.422000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.421906 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" podUID="15f1368b-899a-4ca7-9edd-57b77f4657b5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.106:8443/health\": dial tcp 10.133.0.106:8443: connect: connection refused" Nov 04 01:28:08.609098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.608121 4312 generic.go:356] "Generic (PLEG): container finished" podID="15f1368b-899a-4ca7-9edd-57b77f4657b5" containerID="5157ddb64065f50d2310ab044458bff8ab5da696b1854f23880f5eba2f2508b0" exitCode=0 Nov 04 01:28:08.609098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.608289 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" event={"ID":"15f1368b-899a-4ca7-9edd-57b77f4657b5","Type":"ContainerDied","Data":"5157ddb64065f50d2310ab044458bff8ab5da696b1854f23880f5eba2f2508b0"} Nov 04 01:28:08.782835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.782796 4312 patch_prober.go:29] interesting pod/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 01:28:08.786560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:08.786417 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 01:28:09.030825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.030799 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:28:09.126624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.125375 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-maas-api-tls\") pod \"15f1368b-899a-4ca7-9edd-57b77f4657b5\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " Nov 04 01:28:09.126624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.125449 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-metrics-certs\") pod \"15f1368b-899a-4ca7-9edd-57b77f4657b5\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " Nov 04 01:28:09.126624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.125478 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f1368b-899a-4ca7-9edd-57b77f4657b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f1368b-899a-4ca7-9edd-57b77f4657b5-openshift-service-ca\") pod \"15f1368b-899a-4ca7-9edd-57b77f4657b5\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " Nov 04 01:28:09.126624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.125549 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f1368b-899a-4ca7-9edd-57b77f4657b5-kube-api-access-4lb64\" (UniqueName: \"kubernetes.io/projected/15f1368b-899a-4ca7-9edd-57b77f4657b5-kube-api-access-4lb64\") pod \"15f1368b-899a-4ca7-9edd-57b77f4657b5\" (UID: \"15f1368b-899a-4ca7-9edd-57b77f4657b5\") " Nov 04 01:28:09.128407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.128379 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f1368b-899a-4ca7-9edd-57b77f4657b5-openshift-service-ca" pod "15f1368b-899a-4ca7-9edd-57b77f4657b5" (UID: "15f1368b-899a-4ca7-9edd-57b77f4657b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:09.131364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.131319 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-metrics-certs" pod "15f1368b-899a-4ca7-9edd-57b77f4657b5" (UID: "15f1368b-899a-4ca7-9edd-57b77f4657b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:09.133503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.133254 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:28:09.133714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.133684 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:28:09.135877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.135838 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-maas-api-tls" pod "15f1368b-899a-4ca7-9edd-57b77f4657b5" (UID: "15f1368b-899a-4ca7-9edd-57b77f4657b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:09.141278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.141250 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f1368b-899a-4ca7-9edd-57b77f4657b5-kube-api-access-4lb64" pod "15f1368b-899a-4ca7-9edd-57b77f4657b5" (UID: "15f1368b-899a-4ca7-9edd-57b77f4657b5"). InnerVolumeSpecName "kube-api-access-4lb64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:09.228193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.228154 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:09.228193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.228191 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f1368b-899a-4ca7-9edd-57b77f4657b5-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:09.228515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.228208 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f1368b-899a-4ca7-9edd-57b77f4657b5-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:09.228515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.228235 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lb64\" (UniqueName: \"kubernetes.io/projected/15f1368b-899a-4ca7-9edd-57b77f4657b5-kube-api-access-4lb64\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:09.620127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.620101 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" Nov 04 01:28:09.621425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.620158 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq" event={"ID":"15f1368b-899a-4ca7-9edd-57b77f4657b5","Type":"ContainerDied","Data":"792eb790ee369a879c2f89b87950527169718267c67baf2696f7068b3cfa4eda"} Nov 04 01:28:09.621539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.621464 4312 scope.go:122] "RemoveContainer" containerID="5157ddb64065f50d2310ab044458bff8ab5da696b1854f23880f5eba2f2508b0" Nov 04 01:28:09.673416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.673381 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq"] Nov 04 01:28:09.696645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.696598 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3bf4f9-85bb55b44f-bt4jq"] Nov 04 01:28:09.783513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.783474 4312 patch_prober.go:29] interesting pod/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 01:28:09.784253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:09.784217 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 01:28:10.135833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:10.133381 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:28:10.135833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:10.133455 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:28:10.139026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:10.138994 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f1368b-899a-4ca7-9edd-57b77f4657b5" path="/var/lib/kubelet/pods/15f1368b-899a-4ca7-9edd-57b77f4657b5/volumes" Nov 04 01:28:10.784740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:10.783162 4312 patch_prober.go:29] interesting pod/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 01:28:10.816870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:10.816824 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 01:28:11.132187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:11.130497 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:28:11.132187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:11.130559 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:28:11.797358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:11.778844 4312 patch_prober.go:29] interesting pod/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 01:28:11.797358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:11.778978 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 01:28:12.137408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.136690 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:28:12.137408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.136763 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:28:12.265364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.257388 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:28:12.265364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.259887 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:28:12.518409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.518074 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9"] Nov 04 01:28:12.519241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.519216 4312 cpu_manager.go:414] "containerMap: removing container" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" Nov 04 01:28:12.519443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.519427 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" Nov 04 01:28:12.519561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.519551 4312 cpu_manager.go:414] "containerMap: removing container" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" Nov 04 01:28:12.519665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.519639 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" Nov 04 01:28:12.519869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.519859 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3b40aff9-f6df-482d-b40d-ec023527eb7b" containerName="payload-processing" Nov 04 01:28:12.519946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.519936 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b40aff9-f6df-482d-b40d-ec023527eb7b" containerName="payload-processing" Nov 04 01:28:12.520022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.520013 4312 cpu_manager.go:414] "containerMap: removing container" podUID="15f1368b-899a-4ca7-9edd-57b77f4657b5" containerName="maas-api" Nov 04 01:28:12.520092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.520083 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f1368b-899a-4ca7-9edd-57b77f4657b5" containerName="maas-api" Nov 04 01:28:12.520387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.520373 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b40aff9-f6df-482d-b40d-ec023527eb7b" containerName="payload-processing" Nov 04 01:28:12.520498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.520488 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f1368b-899a-4ca7-9edd-57b77f4657b5" containerName="maas-api" Nov 04 01:28:12.520573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.520565 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" Nov 04 01:28:12.520652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.520643 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f775a2-6d6c-422c-be49-5eca668d52c8" containerName="istio-proxy" Nov 04 01:28:12.542359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.539122 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9"] Nov 04 01:28:12.542359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.539257 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.639904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.639870 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640043 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkv8\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-kube-api-access-fgkv8\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640177 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640212 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640308 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640377 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5357bfb-550e-4296-80f1-939d795a545e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640425 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5357bfb-550e-4296-80f1-939d795a545e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640454 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.640569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.640558 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.741716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.741676 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.741872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.741797 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkv8\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-kube-api-access-fgkv8\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.741872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.741864 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.741980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.741888 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.741980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.741929 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.741980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.741950 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5357bfb-550e-4296-80f1-939d795a545e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.742128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.741987 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5357bfb-550e-4296-80f1-939d795a545e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.742128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.742026 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.742245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.742153 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.742245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.742187 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.742576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.742551 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.742792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.742603 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.743117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.743098 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.743227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.743144 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5357bfb-550e-4296-80f1-939d795a545e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.744970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.744939 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.745587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.745568 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5357bfb-550e-4296-80f1-939d795a545e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.779237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.779204 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkv8\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-kube-api-access-fgkv8\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.783559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.779206 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.785983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.785954 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:28:12.786469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.786445 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:28:12.787239 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.787222 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:28:12.902005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.901975 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:12.930998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.930958 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m"] Nov 04 01:28:12.931115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:12.931053 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd"] Nov 04 01:28:13.131027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.130995 4312 patch_prober.go:29] interesting pod/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:28:13.131178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.131056 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:28:13.276925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.276892 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9"] Nov 04 01:28:13.277620 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:28:13.277587 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5357bfb_550e_4296_80f1_939d795a545e.slice/crio-2a210301188b84fb1b3fc2846b76566ce465be303134b53d08b255b2cbce21bb WatchSource:0}: Error finding container 2a210301188b84fb1b3fc2846b76566ce465be303134b53d08b255b2cbce21bb: Status 404 returned error can't find the container with id 2a210301188b84fb1b3fc2846b76566ce465be303134b53d08b255b2cbce21bb Nov 04 01:28:13.280821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.280788 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:13.280926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.280859 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:13.280926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.280887 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:13.729675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.729579 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" event={"ID":"f5357bfb-550e-4296-80f1-939d795a545e","Type":"ContainerStarted","Data":"5c993d210e8a22120c174836cefe5a648e1ab230bae1b5f08acf17fea92e3ab6"} Nov 04 01:28:13.729875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.729685 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" event={"ID":"f5357bfb-550e-4296-80f1-939d795a545e","Type":"ContainerStarted","Data":"2a210301188b84fb1b3fc2846b76566ce465be303134b53d08b255b2cbce21bb"} Nov 04 01:28:13.773404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.773326 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" podStartSLOduration=1.773305371 podStartE2EDuration="1.773305371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:13.770976334 +0000 UTC m=+1525.793908365" watchObservedRunningTime="2026-11-04 01:28:13.773305371 +0000 UTC m=+1525.796237379" Nov 04 01:28:13.903775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:13.903728 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:14.151367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:14.148818 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:28:14.173730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:14.173695 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:28:14.906359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:14.906268 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 01:28:14.906868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:14.906358 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 01:28:15.911362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:15.903678 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 01:28:15.911362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:15.903737 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 01:28:16.930385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:16.917577 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 01:28:16.930385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:16.917642 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 01:28:17.915358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:17.905846 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 01:28:17.915358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:17.905911 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 01:28:17.942032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:17.941989 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" podUID="6d423e64-8112-41ed-a035-5f15b34cc2fe" containerName="payload-processing" containerID="cri-o://40389157424c2702caf1dd2ca5ca53b5e2049b63d323ee196eed73096a810306" gracePeriod=25 Nov 04 01:28:17.942817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:17.942632 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" podUID="03d9433c-03e4-4a44-889f-bed3f3c8a24c" containerName="payload-pre-processing" containerID="cri-o://eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4" gracePeriod=25 Nov 04 01:28:18.840512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:18.839739 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:28:18.909404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:18.908745 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.909404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:18.908823 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 04 01:28:19.038789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.023688 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d9433c-03e4-4a44-889f-bed3f3c8a24c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d9433c-03e4-4a44-889f-bed3f3c8a24c-plugins-config-volume\") pod \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " Nov 04 01:28:19.038789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.037605 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d9433c-03e4-4a44-889f-bed3f3c8a24c-kube-api-access-mg2dt\" (UniqueName: \"kubernetes.io/projected/03d9433c-03e4-4a44-889f-bed3f3c8a24c-kube-api-access-mg2dt\") pod \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\" (UID: \"03d9433c-03e4-4a44-889f-bed3f3c8a24c\") " Nov 04 01:28:19.038789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.034561 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d9433c-03e4-4a44-889f-bed3f3c8a24c-plugins-config-volume" pod "03d9433c-03e4-4a44-889f-bed3f3c8a24c" (UID: "03d9433c-03e4-4a44-889f-bed3f3c8a24c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:19.063421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.047266 4312 generic.go:356] "Generic (PLEG): container finished" podID="03d9433c-03e4-4a44-889f-bed3f3c8a24c" containerID="eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4" exitCode=0 Nov 04 01:28:19.063421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.047431 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" event={"ID":"03d9433c-03e4-4a44-889f-bed3f3c8a24c","Type":"ContainerDied","Data":"eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4"} Nov 04 01:28:19.063421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.047507 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" event={"ID":"03d9433c-03e4-4a44-889f-bed3f3c8a24c","Type":"ContainerDied","Data":"f17fe76a48e9d4c83d8dc7501d876b9836704fc032a95ad17685d448846e497b"} Nov 04 01:28:19.063421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.047540 4312 scope.go:122] "RemoveContainer" containerID="eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4" Nov 04 01:28:19.063421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.051612 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd" Nov 04 01:28:19.092515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.091123 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d9433c-03e4-4a44-889f-bed3f3c8a24c-kube-api-access-mg2dt" pod "03d9433c-03e4-4a44-889f-bed3f3c8a24c" (UID: "03d9433c-03e4-4a44-889f-bed3f3c8a24c"). InnerVolumeSpecName "kube-api-access-mg2dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:19.142003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.141912 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg2dt\" (UniqueName: \"kubernetes.io/projected/03d9433c-03e4-4a44-889f-bed3f3c8a24c-kube-api-access-mg2dt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:19.142003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.141952 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d9433c-03e4-4a44-889f-bed3f3c8a24c-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:19.576851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.576718 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9"] Nov 04 01:28:19.577391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.577358 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" containerID="cri-o://5c993d210e8a22120c174836cefe5a648e1ab230bae1b5f08acf17fea92e3ab6" gracePeriod=30 Nov 04 01:28:19.694533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.694497 4312 generic.go:356] "Generic (PLEG): container finished" podID="6d423e64-8112-41ed-a035-5f15b34cc2fe" containerID="40389157424c2702caf1dd2ca5ca53b5e2049b63d323ee196eed73096a810306" exitCode=0 Nov 04 01:28:19.695005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.694981 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" event={"ID":"6d423e64-8112-41ed-a035-5f15b34cc2fe","Type":"ContainerDied","Data":"40389157424c2702caf1dd2ca5ca53b5e2049b63d323ee196eed73096a810306"} Nov 04 01:28:19.803360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.799357 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f"] Nov 04 01:28:19.803360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.799847 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" containerID="cri-o://17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e" gracePeriod=30 Nov 04 01:28:19.807664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.807621 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:28:20.016241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.934521 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg"] Nov 04 01:28:20.016889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.016847 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" containerID="cri-o://37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09" gracePeriod=30 Nov 04 01:28:20.017175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:19.989675 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d423e64-8112-41ed-a035-5f15b34cc2fe-kube-api-access-lsbrx\" (UniqueName: \"kubernetes.io/projected/6d423e64-8112-41ed-a035-5f15b34cc2fe-kube-api-access-lsbrx\") pod \"6d423e64-8112-41ed-a035-5f15b34cc2fe\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " Nov 04 01:28:20.017469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.017446 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d423e64-8112-41ed-a035-5f15b34cc2fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d423e64-8112-41ed-a035-5f15b34cc2fe-plugins-config-volume\") pod \"6d423e64-8112-41ed-a035-5f15b34cc2fe\" (UID: \"6d423e64-8112-41ed-a035-5f15b34cc2fe\") " Nov 04 01:28:20.026360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.024491 4312 scope.go:122] "RemoveContainer" containerID="eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4" Nov 04 01:28:20.044367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.030035 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d423e64-8112-41ed-a035-5f15b34cc2fe-kube-api-access-lsbrx" pod "6d423e64-8112-41ed-a035-5f15b34cc2fe" (UID: "6d423e64-8112-41ed-a035-5f15b34cc2fe"). InnerVolumeSpecName "kube-api-access-lsbrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:20.044367 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:20.035491 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4\": container with ID starting with eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4 not found: ID does not exist" containerID="eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4" Nov 04 01:28:20.044367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.035532 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4"} err="failed to get container status \"eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4\": rpc error: code = NotFound desc = could not find container \"eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4\": container with ID starting with eca630ca61102e46134b2d6a772f016db075a7114df48d8f8fe5576b542355e4 not found: ID does not exist" Nov 04 01:28:20.065369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.045287 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d423e64-8112-41ed-a035-5f15b34cc2fe-plugins-config-volume" pod "6d423e64-8112-41ed-a035-5f15b34cc2fe" (UID: "6d423e64-8112-41ed-a035-5f15b34cc2fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:20.204539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.152205 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsbrx\" (UniqueName: \"kubernetes.io/projected/6d423e64-8112-41ed-a035-5f15b34cc2fe-kube-api-access-lsbrx\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:20.204539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.203722 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d423e64-8112-41ed-a035-5f15b34cc2fe-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:20.268382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.237305 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:28:20.393896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.390708 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd"] Nov 04 01:28:20.393896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.390761 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d2e4e3-946dc468c-qqmzd"] Nov 04 01:28:20.393896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.390787 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t"] Nov 04 01:28:20.393896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.390835 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms"] Nov 04 01:28:20.427371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.427170 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:28:20.443227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.443190 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:20.721545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.721508 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" Nov 04 01:28:20.721545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.721531 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m" event={"ID":"6d423e64-8112-41ed-a035-5f15b34cc2fe","Type":"ContainerDied","Data":"a4ed1a6a16eab63fe3d8f92e7acc9675f2a2519bed6dd644423c7227b31dd0b3"} Nov 04 01:28:20.721768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.721633 4312 scope.go:122] "RemoveContainer" containerID="40389157424c2702caf1dd2ca5ca53b5e2049b63d323ee196eed73096a810306" Nov 04 01:28:20.746035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.746011 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m"] Nov 04 01:28:20.765468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:20.765424 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d2e4e3-64c4ffddc6-szw6m"] Nov 04 01:28:22.113720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:22.113690 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d9433c-03e4-4a44-889f-bed3f3c8a24c" path="/var/lib/kubelet/pods/03d9433c-03e4-4a44-889f-bed3f3c8a24c/volumes" Nov 04 01:28:22.114376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:22.114325 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d423e64-8112-41ed-a035-5f15b34cc2fe" path="/var/lib/kubelet/pods/6d423e64-8112-41ed-a035-5f15b34cc2fe/volumes" Nov 04 01:28:23.691611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:23.691567 4312 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:23.691986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:23.691638 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:25.222204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.221871 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:28:25.267965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.267858 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-certs\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.277430 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-certs" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.277542 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-credential-socket\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.277650 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-token\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.277761 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-data\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.277864 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-kube-api-access-ctfds\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-kube-api-access-ctfds\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.277908 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-socket\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.277999 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istiod-ca-cert\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.278064 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-envoy\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.278124 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-podinfo\") pod \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\" (UID: \"c5b6ab23-7925-4dc8-a773-f70db1a2fe06\") " Nov 04 01:28:25.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.279009 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.281045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.280729 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istiod-ca-cert" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:25.281045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.280988 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-socket" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.284395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.281161 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-data" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.284395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.281327 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-credential-socket" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.287406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.287361 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-podinfo" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:25.287547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.287527 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-token" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:25.287651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.287631 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-envoy" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.287767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.287670 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-kube-api-access-ctfds" pod "c5b6ab23-7925-4dc8-a773-f70db1a2fe06" (UID: "c5b6ab23-7925-4dc8-a773-f70db1a2fe06"). InnerVolumeSpecName "kube-api-access-ctfds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:25.382854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382784 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.382854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382829 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.382854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382847 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.382854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382860 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.383118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382904 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfds\" (UniqueName: \"kubernetes.io/projected/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-kube-api-access-ctfds\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.383118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382920 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.383118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382934 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.383118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.382947 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6ab23-7925-4dc8-a773-f70db1a2fe06-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.391494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.391455 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" podUID="51241f78-295d-4d43-8ed0-dab0ba0e7503" containerName="payload-pre-processing" containerID="cri-o://4c213262b658b7f6baa964655b4f2ba2b35d2eefd9447d889aabc426478adcb5" gracePeriod=25 Nov 04 01:28:25.393503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.393439 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" podUID="2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" containerName="payload-processing" containerID="cri-o://981db677fb7d4f70d270aa98af9fd5caedb988b1f3aec3fd70ca3d323ebf7e6e" gracePeriod=25 Nov 04 01:28:25.513939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.513911 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586029 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-token\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-token\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586107 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-certs\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586143 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-kube-api-access-xlwf7\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-kube-api-access-xlwf7\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586170 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-podinfo\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586220 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-envoy\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586292 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-socket\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586353 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-credential-socket\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586435 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5ed29b-ec80-4bef-bc71-4c185119193a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5ed29b-ec80-4bef-bc71-4c185119193a-istiod-ca-cert\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.587576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.586462 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-data\") pod \"ee5ed29b-ec80-4bef-bc71-4c185119193a\" (UID: \"ee5ed29b-ec80-4bef-bc71-4c185119193a\") " Nov 04 01:28:25.590878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.587719 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-credential-socket" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.590878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.587742 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-data" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.590878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.587930 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-socket" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.590878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.588101 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5ed29b-ec80-4bef-bc71-4c185119193a-istiod-ca-cert" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:25.590878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.588394 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-certs" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.590878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.590747 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-envoy" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:25.591189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.590893 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-token" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:25.592665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.592634 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-kube-api-access-xlwf7" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "kube-api-access-xlwf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:25.602466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.602405 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-podinfo" pod "ee5ed29b-ec80-4bef-bc71-4c185119193a" (UID: "ee5ed29b-ec80-4bef-bc71-4c185119193a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:25.689204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689132 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689162 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689177 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlwf7\" (UniqueName: \"kubernetes.io/projected/ee5ed29b-ec80-4bef-bc71-4c185119193a-kube-api-access-xlwf7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689190 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689202 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689215 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689227 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689239 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5ed29b-ec80-4bef-bc71-4c185119193a-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.689691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.689251 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed29b-ec80-4bef-bc71-4c185119193a-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.803022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.801087 4312 generic.go:356] "Generic (PLEG): container finished" podID="2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" containerID="981db677fb7d4f70d270aa98af9fd5caedb988b1f3aec3fd70ca3d323ebf7e6e" exitCode=0 Nov 04 01:28:25.803022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.801194 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" event={"ID":"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759","Type":"ContainerDied","Data":"981db677fb7d4f70d270aa98af9fd5caedb988b1f3aec3fd70ca3d323ebf7e6e"} Nov 04 01:28:25.803721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.803695 4312 generic.go:356] "Generic (PLEG): container finished" podID="51241f78-295d-4d43-8ed0-dab0ba0e7503" containerID="4c213262b658b7f6baa964655b4f2ba2b35d2eefd9447d889aabc426478adcb5" exitCode=0 Nov 04 01:28:25.804013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.803837 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" event={"ID":"51241f78-295d-4d43-8ed0-dab0ba0e7503","Type":"ContainerDied","Data":"4c213262b658b7f6baa964655b4f2ba2b35d2eefd9447d889aabc426478adcb5"} Nov 04 01:28:25.805210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.805185 4312 generic.go:356] "Generic (PLEG): container finished" podID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerID="37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09" exitCode=0 Nov 04 01:28:25.805364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.805349 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" Nov 04 01:28:25.806438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.806409 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" event={"ID":"ee5ed29b-ec80-4bef-bc71-4c185119193a","Type":"ContainerDied","Data":"37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09"} Nov 04 01:28:25.806542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.806468 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg" event={"ID":"ee5ed29b-ec80-4bef-bc71-4c185119193a","Type":"ContainerDied","Data":"41fff653bc9b86f2ca8abc5ee20955dc40d46fc3f5c4c20d1242c0713eb8a592"} Nov 04 01:28:25.806542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.806498 4312 scope.go:122] "RemoveContainer" containerID="37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09" Nov 04 01:28:25.811834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.811812 4312 generic.go:356] "Generic (PLEG): container finished" podID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerID="17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e" exitCode=0 Nov 04 01:28:25.811921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.811907 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" event={"ID":"c5b6ab23-7925-4dc8-a773-f70db1a2fe06","Type":"ContainerDied","Data":"17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e"} Nov 04 01:28:25.811992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.811951 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" event={"ID":"c5b6ab23-7925-4dc8-a773-f70db1a2fe06","Type":"ContainerDied","Data":"32555fcf8c718d6c1798acc6b0e08f03a06ec3f6b4b1ba4446a137ed27f10048"} Nov 04 01:28:25.812057 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.812049 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f" Nov 04 01:28:25.880617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.880591 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:28:25.881927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.881903 4312 scope.go:122] "RemoveContainer" containerID="37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09" Nov 04 01:28:25.882226 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:25.882203 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09\": container with ID starting with 37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09 not found: ID does not exist" containerID="37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09" Nov 04 01:28:25.882315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.882236 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09"} err="failed to get container status \"37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09\": rpc error: code = NotFound desc = could not find container \"37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09\": container with ID starting with 37ceb27954185ede65d0b7c9b8d9f4ffaff035866deeeff8ca72ff7d958aaa09 not found: ID does not exist" Nov 04 01:28:25.882315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.882253 4312 scope.go:122] "RemoveContainer" containerID="17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e" Nov 04 01:28:25.887222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.887204 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:28:25.894094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.894075 4312 scope.go:122] "RemoveContainer" containerID="17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e" Nov 04 01:28:25.894382 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:25.894355 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e\": container with ID starting with 17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e not found: ID does not exist" containerID="17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e" Nov 04 01:28:25.894481 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.894391 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e"} err="failed to get container status \"17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e\": rpc error: code = NotFound desc = could not find container \"17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e\": container with ID starting with 17f367dd2139ee5ebaa4d3a8086ecbe4095f9f9a801961702841874e25702c2e not found: ID does not exist" Nov 04 01:28:25.897273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.897253 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg"] Nov 04 01:28:25.903455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.903435 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-hfrlg"] Nov 04 01:28:25.916825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.916782 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f"] Nov 04 01:28:25.921684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.921662 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-rgd4f"] Nov 04 01:28:25.994403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.994286 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-plugins-config-volume\") pod \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " Nov 04 01:28:25.994403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.994376 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51241f78-295d-4d43-8ed0-dab0ba0e7503-kube-api-access-6ww42\" (UniqueName: \"kubernetes.io/projected/51241f78-295d-4d43-8ed0-dab0ba0e7503-kube-api-access-6ww42\") pod \"51241f78-295d-4d43-8ed0-dab0ba0e7503\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " Nov 04 01:28:25.994635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.994463 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51241f78-295d-4d43-8ed0-dab0ba0e7503-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51241f78-295d-4d43-8ed0-dab0ba0e7503-plugins-config-volume\") pod \"51241f78-295d-4d43-8ed0-dab0ba0e7503\" (UID: \"51241f78-295d-4d43-8ed0-dab0ba0e7503\") " Nov 04 01:28:25.994949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.994926 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-plugins-config-volume" pod "2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" (UID: "2138bbdb-3ef7-4ad4-83d5-ac22fabcd759"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:25.995436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.995406 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51241f78-295d-4d43-8ed0-dab0ba0e7503-plugins-config-volume" pod "51241f78-295d-4d43-8ed0-dab0ba0e7503" (UID: "51241f78-295d-4d43-8ed0-dab0ba0e7503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:25.996431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.996406 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-kube-api-access-p5h62\" (UniqueName: \"kubernetes.io/projected/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-kube-api-access-p5h62\") pod \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\" (UID: \"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759\") " Nov 04 01:28:25.997168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.997141 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:25.997245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:25.997173 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51241f78-295d-4d43-8ed0-dab0ba0e7503-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:26.001704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.000834 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51241f78-295d-4d43-8ed0-dab0ba0e7503-kube-api-access-6ww42" pod "51241f78-295d-4d43-8ed0-dab0ba0e7503" (UID: "51241f78-295d-4d43-8ed0-dab0ba0e7503"). InnerVolumeSpecName "kube-api-access-6ww42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:26.002704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.002675 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-kube-api-access-p5h62" pod "2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" (UID: "2138bbdb-3ef7-4ad4-83d5-ac22fabcd759"). InnerVolumeSpecName "kube-api-access-p5h62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:26.098457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.098425 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5h62\" (UniqueName: \"kubernetes.io/projected/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759-kube-api-access-p5h62\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:26.098457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.098457 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ww42\" (UniqueName: \"kubernetes.io/projected/51241f78-295d-4d43-8ed0-dab0ba0e7503-kube-api-access-6ww42\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:26.111618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.111579 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" path="/var/lib/kubelet/pods/c5b6ab23-7925-4dc8-a773-f70db1a2fe06/volumes" Nov 04 01:28:26.112288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.112264 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" path="/var/lib/kubelet/pods/ee5ed29b-ec80-4bef-bc71-4c185119193a/volumes" Nov 04 01:28:26.212235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.212199 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz"] Nov 04 01:28:26.212509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.212478 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" podUID="64548438-f3dd-488f-a623-9cd17e97af8e" containerName="maas-api" containerID="cri-o://70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90" gracePeriod=30 Nov 04 01:28:26.594885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.594852 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:28:26.747875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.747846 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64548438-f3dd-488f-a623-9cd17e97af8e-kube-api-access-vgjft\" (UniqueName: \"kubernetes.io/projected/64548438-f3dd-488f-a623-9cd17e97af8e-kube-api-access-vgjft\") pod \"64548438-f3dd-488f-a623-9cd17e97af8e\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " Nov 04 01:28:26.748024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.748012 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64548438-f3dd-488f-a623-9cd17e97af8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64548438-f3dd-488f-a623-9cd17e97af8e-openshift-service-ca\") pod \"64548438-f3dd-488f-a623-9cd17e97af8e\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " Nov 04 01:28:26.748137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.748124 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-maas-api-tls\") pod \"64548438-f3dd-488f-a623-9cd17e97af8e\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " Nov 04 01:28:26.748207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.748157 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-metrics-certs\") pod \"64548438-f3dd-488f-a623-9cd17e97af8e\" (UID: \"64548438-f3dd-488f-a623-9cd17e97af8e\") " Nov 04 01:28:26.748448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.748412 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64548438-f3dd-488f-a623-9cd17e97af8e-openshift-service-ca" pod "64548438-f3dd-488f-a623-9cd17e97af8e" (UID: "64548438-f3dd-488f-a623-9cd17e97af8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:26.748774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.748754 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64548438-f3dd-488f-a623-9cd17e97af8e-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:26.749999 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.749978 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64548438-f3dd-488f-a623-9cd17e97af8e-kube-api-access-vgjft" pod "64548438-f3dd-488f-a623-9cd17e97af8e" (UID: "64548438-f3dd-488f-a623-9cd17e97af8e"). InnerVolumeSpecName "kube-api-access-vgjft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:26.750580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.750556 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-maas-api-tls" pod "64548438-f3dd-488f-a623-9cd17e97af8e" (UID: "64548438-f3dd-488f-a623-9cd17e97af8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:26.750673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.750620 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-metrics-certs" pod "64548438-f3dd-488f-a623-9cd17e97af8e" (UID: "64548438-f3dd-488f-a623-9cd17e97af8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:26.826128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.826047 4312 generic.go:356] "Generic (PLEG): container finished" podID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerID="af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa" exitCode=0 Nov 04 01:28:26.826267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.826117 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" event={"ID":"3379949f-d9f1-4c2e-9341-407cbdfc7b85","Type":"ContainerDied","Data":"af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa"} Nov 04 01:28:26.827396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.827347 4312 scope.go:122] "RemoveContainer" containerID="af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa" Nov 04 01:28:26.827904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.827883 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" Nov 04 01:28:26.828026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.827897 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t" event={"ID":"2138bbdb-3ef7-4ad4-83d5-ac22fabcd759","Type":"ContainerDied","Data":"2b9697345d21b4ce2060eff9654cef3627d817211d197d643f8288be73001a08"} Nov 04 01:28:26.828026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.827955 4312 scope.go:122] "RemoveContainer" containerID="981db677fb7d4f70d270aa98af9fd5caedb988b1f3aec3fd70ca3d323ebf7e6e" Nov 04 01:28:26.829906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.829882 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" event={"ID":"51241f78-295d-4d43-8ed0-dab0ba0e7503","Type":"ContainerDied","Data":"e557285a53433582ccda717f2d5e77bde847123c324d0e9d1cd9b0c24f64a2e5"} Nov 04 01:28:26.830036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.829957 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms" Nov 04 01:28:26.830036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.829976 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:26.830163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.830063 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:26.830163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.830112 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:26.835073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.835046 4312 generic.go:356] "Generic (PLEG): container finished" podID="f5357bfb-550e-4296-80f1-939d795a545e" containerID="5c993d210e8a22120c174836cefe5a648e1ab230bae1b5f08acf17fea92e3ab6" exitCode=0 Nov 04 01:28:26.835247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.835180 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" event={"ID":"f5357bfb-550e-4296-80f1-939d795a545e","Type":"ContainerDied","Data":"5c993d210e8a22120c174836cefe5a648e1ab230bae1b5f08acf17fea92e3ab6"} Nov 04 01:28:26.838455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.838429 4312 generic.go:356] "Generic (PLEG): container finished" podID="873797b0-f49c-467f-8157-1fccaf30b023" containerID="d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea" exitCode=0 Nov 04 01:28:26.842915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.842891 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" event={"ID":"873797b0-f49c-467f-8157-1fccaf30b023","Type":"ContainerDied","Data":"d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea"} Nov 04 01:28:26.843550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.843530 4312 scope.go:122] "RemoveContainer" containerID="d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea" Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.848192 4312 generic.go:356] "Generic (PLEG): container finished" podID="64548438-f3dd-488f-a623-9cd17e97af8e" containerID="70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90" exitCode=0 Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.848243 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" event={"ID":"64548438-f3dd-488f-a623-9cd17e97af8e","Type":"ContainerDied","Data":"70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90"} Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.848282 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" event={"ID":"64548438-f3dd-488f-a623-9cd17e97af8e","Type":"ContainerDied","Data":"6f405901d844f3445e342f72dc6d072ae1266adfe3f6690fa4292f20bd8ddd5a"} Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.848402 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz" Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.855524 4312 scope.go:122] "RemoveContainer" containerID="4c213262b658b7f6baa964655b4f2ba2b35d2eefd9447d889aabc426478adcb5" Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.856449 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.856465 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64548438-f3dd-488f-a623-9cd17e97af8e-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.856477 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgjft\" (UniqueName: \"kubernetes.io/projected/64548438-f3dd-488f-a623-9cd17e97af8e-kube-api-access-vgjft\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.856542 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.856635 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:26.860453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.856685 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:26.892429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.892409 4312 scope.go:122] "RemoveContainer" containerID="70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90" Nov 04 01:28:26.909470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.909379 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms"] Nov 04 01:28:26.916705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.916680 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26f3d2-68fd46846c-qpzms"] Nov 04 01:28:26.932085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.932049 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t"] Nov 04 01:28:26.937691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.937666 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26f3d2-96b7b46c6-lf56t"] Nov 04 01:28:26.949543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.949529 4312 scope.go:122] "RemoveContainer" containerID="70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90" Nov 04 01:28:26.949936 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:26.949909 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90\": container with ID starting with 70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90 not found: ID does not exist" containerID="70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90" Nov 04 01:28:26.950021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.949948 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90"} err="failed to get container status \"70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90\": rpc error: code = NotFound desc = could not find container \"70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90\": container with ID starting with 70016ad829afdb022bbd780940383614a2be45788c14fa09ffcafa6cf1cc8d90 not found: ID does not exist" Nov 04 01:28:26.974021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.973994 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz"] Nov 04 01:28:26.977132 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:26.977108 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2e4e3-855fcbff4d-kxwjz"] Nov 04 01:28:27.065979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.065947 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:27.162055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.161933 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5357bfb-550e-4296-80f1-939d795a545e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5357bfb-550e-4296-80f1-939d795a545e-istio-podinfo\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162006 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-kube-api-access-fgkv8\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-kube-api-access-fgkv8\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162050 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5357bfb-550e-4296-80f1-939d795a545e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5357bfb-550e-4296-80f1-939d795a545e-istiod-ca-cert\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162198 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-certs\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162270 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-envoy\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162344 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162320 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-istio-token\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-istio-token\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162417 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-credential-socket\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162472 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-data\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162525 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-socket\") pod \"f5357bfb-550e-4296-80f1-939d795a545e\" (UID: \"f5357bfb-550e-4296-80f1-939d795a545e\") " Nov 04 01:28:27.162902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.162732 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5357bfb-550e-4296-80f1-939d795a545e-istiod-ca-cert" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:27.163257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.163158 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-certs" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:27.163257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.163188 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5357bfb-550e-4296-80f1-939d795a545e-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.163392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.163287 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-data" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:27.163547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.163523 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-socket" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:27.163632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.163604 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-credential-socket" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:27.165891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.165859 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-kube-api-access-fgkv8" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "kube-api-access-fgkv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:27.166007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.165896 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-istio-token" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:27.166007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.165985 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-envoy" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:27.166485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.166461 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5357bfb-550e-4296-80f1-939d795a545e-istio-podinfo" pod "f5357bfb-550e-4296-80f1-939d795a545e" (UID: "f5357bfb-550e-4296-80f1-939d795a545e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:27.209637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.209604 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z"] Nov 04 01:28:27.210128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.209869 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" podUID="2b81fcbb-dfaf-467c-9da0-913c1f77a907" containerName="maas-api" containerID="cri-o://a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f" gracePeriod=30 Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265649 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265685 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265701 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265715 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265738 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265752 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5357bfb-550e-4296-80f1-939d795a545e-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265766 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgkv8\" (UniqueName: \"kubernetes.io/projected/f5357bfb-550e-4296-80f1-939d795a545e-kube-api-access-fgkv8\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.268743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.265781 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5357bfb-550e-4296-80f1-939d795a545e-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.620820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.620791 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:28:27.776480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.775754 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-metrics-certs\") pod \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " Nov 04 01:28:27.776480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.775851 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b81fcbb-dfaf-467c-9da0-913c1f77a907-kube-api-access-rfljh\" (UniqueName: \"kubernetes.io/projected/2b81fcbb-dfaf-467c-9da0-913c1f77a907-kube-api-access-rfljh\") pod \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " Nov 04 01:28:27.776480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.775970 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b81fcbb-dfaf-467c-9da0-913c1f77a907-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b81fcbb-dfaf-467c-9da0-913c1f77a907-openshift-service-ca\") pod \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " Nov 04 01:28:27.776480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.776132 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-maas-api-tls\") pod \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\" (UID: \"2b81fcbb-dfaf-467c-9da0-913c1f77a907\") " Nov 04 01:28:27.776480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.776387 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b81fcbb-dfaf-467c-9da0-913c1f77a907-openshift-service-ca" pod "2b81fcbb-dfaf-467c-9da0-913c1f77a907" (UID: "2b81fcbb-dfaf-467c-9da0-913c1f77a907"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:27.776841 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.776815 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b81fcbb-dfaf-467c-9da0-913c1f77a907-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.778735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.778708 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-maas-api-tls" pod "2b81fcbb-dfaf-467c-9da0-913c1f77a907" (UID: "2b81fcbb-dfaf-467c-9da0-913c1f77a907"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:27.778853 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.778755 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b81fcbb-dfaf-467c-9da0-913c1f77a907-kube-api-access-rfljh" pod "2b81fcbb-dfaf-467c-9da0-913c1f77a907" (UID: "2b81fcbb-dfaf-467c-9da0-913c1f77a907"). InnerVolumeSpecName "kube-api-access-rfljh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:27.778931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.778914 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-metrics-certs" pod "2b81fcbb-dfaf-467c-9da0-913c1f77a907" (UID: "2b81fcbb-dfaf-467c-9da0-913c1f77a907"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:27.891375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.887731 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.891375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.887771 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b81fcbb-dfaf-467c-9da0-913c1f77a907-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.891375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.887787 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfljh\" (UniqueName: \"kubernetes.io/projected/2b81fcbb-dfaf-467c-9da0-913c1f77a907-kube-api-access-rfljh\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:27.922365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.922304 4312 generic.go:356] "Generic (PLEG): container finished" podID="2b81fcbb-dfaf-467c-9da0-913c1f77a907" containerID="a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f" exitCode=0 Nov 04 01:28:27.922549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.922500 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" Nov 04 01:28:27.938570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.938532 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" event={"ID":"2b81fcbb-dfaf-467c-9da0-913c1f77a907","Type":"ContainerDied","Data":"a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f"} Nov 04 01:28:27.940764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.940733 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z" event={"ID":"2b81fcbb-dfaf-467c-9da0-913c1f77a907","Type":"ContainerDied","Data":"bfaeaf2a2fa64d992114919a5dd53180fe5cd18e7d0122eb93ee8c2f11937943"} Nov 04 01:28:27.940983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:27.940965 4312 scope.go:122] "RemoveContainer" containerID="a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f" Nov 04 01:28:28.040864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.039914 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" Nov 04 01:28:28.041164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.041135 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9" event={"ID":"f5357bfb-550e-4296-80f1-939d795a545e","Type":"ContainerDied","Data":"2a210301188b84fb1b3fc2846b76566ce465be303134b53d08b255b2cbce21bb"} Nov 04 01:28:28.145057 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.145025 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" path="/var/lib/kubelet/pods/2138bbdb-3ef7-4ad4-83d5-ac22fabcd759/volumes" Nov 04 01:28:28.145733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.145705 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51241f78-295d-4d43-8ed0-dab0ba0e7503" path="/var/lib/kubelet/pods/51241f78-295d-4d43-8ed0-dab0ba0e7503/volumes" Nov 04 01:28:28.146304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.146280 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64548438-f3dd-488f-a623-9cd17e97af8e" path="/var/lib/kubelet/pods/64548438-f3dd-488f-a623-9cd17e97af8e/volumes" Nov 04 01:28:28.146711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.146686 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" event={"ID":"873797b0-f49c-467f-8157-1fccaf30b023","Type":"ContainerStarted","Data":"a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0"} Nov 04 01:28:28.146864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.146738 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z"] Nov 04 01:28:28.146864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.146764 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" event={"ID":"3379949f-d9f1-4c2e-9341-407cbdfc7b85","Type":"ContainerStarted","Data":"a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852"} Nov 04 01:28:28.162376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.161149 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26f3d2-847994b4db-lf26z"] Nov 04 01:28:28.170444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.170069 4312 scope.go:122] "RemoveContainer" containerID="a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f" Nov 04 01:28:28.171058 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:28.170691 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f\": container with ID starting with a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f not found: ID does not exist" containerID="a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f" Nov 04 01:28:28.171058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.170731 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f"} err="failed to get container status \"a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f\": rpc error: code = NotFound desc = could not find container \"a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f\": container with ID starting with a8780f4b31108a5a9408fdf0b0739542816c65e7baf5483475644581a194a00f not found: ID does not exist" Nov 04 01:28:28.171058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.170757 4312 scope.go:122] "RemoveContainer" containerID="5c993d210e8a22120c174836cefe5a648e1ab230bae1b5f08acf17fea92e3ab6" Nov 04 01:28:28.224051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.223990 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9"] Nov 04 01:28:28.228799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.228755 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-bwmt9"] Nov 04 01:28:28.614079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.614047 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:28:28.615556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.615065 4312 patch_prober.go:29] interesting pod/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 01:28:28.615754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.615732 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 01:28:28.728641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.728557 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:28:28.730146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.730119 4312 patch_prober.go:29] interesting pod/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 04 01:28:28.730274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:28.730169 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 04 01:28:29.500507 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:29.500470 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:28:29.614245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:29.614203 4312 patch_prober.go:29] interesting pod/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 01:28:29.614441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:29.614261 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 01:28:29.727630 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:29.727587 4312 patch_prober.go:29] interesting pod/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 04 01:28:29.727808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:29.727645 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 04 01:28:30.115772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:30.115736 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b81fcbb-dfaf-467c-9da0-913c1f77a907" path="/var/lib/kubelet/pods/2b81fcbb-dfaf-467c-9da0-913c1f77a907/volumes" Nov 04 01:28:30.116986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:30.116960 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5357bfb-550e-4296-80f1-939d795a545e" path="/var/lib/kubelet/pods/f5357bfb-550e-4296-80f1-939d795a545e/volumes" Nov 04 01:28:30.613993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:30.613947 4312 patch_prober.go:29] interesting pod/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 01:28:30.614199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:30.614044 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 01:28:30.731978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:30.731944 4312 patch_prober.go:29] interesting pod/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 04 01:28:30.732254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:30.732215 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 04 01:28:31.614959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:31.614923 4312 patch_prober.go:29] interesting pod/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 01:28:31.615616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:31.615550 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 01:28:31.728279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:31.728243 4312 patch_prober.go:29] interesting pod/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 04 01:28:31.728464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:31.728318 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 04 01:28:32.424472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:32.424432 4312 patch_prober.go:29] interesting pod/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:32.424661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:32.424486 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:32.614009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:32.613964 4312 patch_prober.go:29] interesting pod/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 01:28:32.614188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:32.614045 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 01:28:32.728320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:32.728242 4312 patch_prober.go:29] interesting pod/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 04 01:28:32.728320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:32.728295 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 04 01:28:33.351361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:33.348467 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:28:33.351361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:33.349458 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:28:33.614886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:33.614800 4312 patch_prober.go:29] interesting pod/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 01:28:33.615040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:33.614896 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 01:28:33.857035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:33.856998 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:28:33.859526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:33.859503 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:28:34.546084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.546049 4312 generic.go:356] "Generic (PLEG): container finished" podID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerID="b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9" exitCode=0 Nov 04 01:28:34.546265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.546118 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" event={"ID":"593b2775-f312-4b74-9c40-c0ab0fdf5b7d","Type":"ContainerDied","Data":"b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9"} Nov 04 01:28:34.546823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.546797 4312 scope.go:122] "RemoveContainer" containerID="b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9" Nov 04 01:28:34.548363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.548319 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:34.548432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.548404 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:34.548468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.548438 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:28:34.620117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.620079 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:28:34.621788 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.621765 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:28:34.765494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:34.765458 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9"] Nov 04 01:28:35.559196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:35.559164 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" event={"ID":"593b2775-f312-4b74-9c40-c0ab0fdf5b7d","Type":"ContainerStarted","Data":"20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef"} Nov 04 01:28:35.559663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:35.559230 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" containerID="cri-o://20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef" gracePeriod=30 Nov 04 01:28:39.065787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:39.065751 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64"] Nov 04 01:28:39.076616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:39.076587 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv"] Nov 04 01:28:40.847634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.847607 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:28:40.982076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982041 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-podinfo\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982103 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-certs\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982168 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-socket\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982217 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-data\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982243 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istiod-ca-cert\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982312 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-kube-api-access-9dvdw\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-kube-api-access-9dvdw\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982348 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-token\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-token\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982422 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-credential-socket\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982500 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-certs" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:40.982711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982509 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-envoy\") pod \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\" (UID: \"593b2775-f312-4b74-9c40-c0ab0fdf5b7d\") " Nov 04 01:28:40.982711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982536 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-socket" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:40.982804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982769 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-data" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:40.982804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982786 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-credential-socket" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:40.982906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.982828 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istiod-ca-cert" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:40.983254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.983232 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:40.983254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.983253 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:40.983462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.983262 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:40.983462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.983271 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:40.983462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.983279 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:40.984477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.984451 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-podinfo" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:40.984992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.984967 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-token" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:40.985109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.985017 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-envoy" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:40.985167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:40.985109 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-kube-api-access-9dvdw" pod "593b2775-f312-4b74-9c40-c0ab0fdf5b7d" (UID: "593b2775-f312-4b74-9c40-c0ab0fdf5b7d"). InnerVolumeSpecName "kube-api-access-9dvdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:41.084325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.084250 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:41.084325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.084279 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:41.084325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.084289 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dvdw\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-kube-api-access-9dvdw\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:41.084325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.084300 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593b2775-f312-4b74-9c40-c0ab0fdf5b7d-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:41.609514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.609473 4312 generic.go:356] "Generic (PLEG): container finished" podID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerID="20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef" exitCode=0 Nov 04 01:28:41.609682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.609523 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" Nov 04 01:28:41.609682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.609596 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" event={"ID":"593b2775-f312-4b74-9c40-c0ab0fdf5b7d","Type":"ContainerDied","Data":"20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef"} Nov 04 01:28:41.609682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.609653 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9" event={"ID":"593b2775-f312-4b74-9c40-c0ab0fdf5b7d","Type":"ContainerDied","Data":"baeaa19a1fc299441ebfc846d6a95c23d4a79d2e228543871f8b86b1e12bc6a5"} Nov 04 01:28:41.609682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.609680 4312 scope.go:122] "RemoveContainer" containerID="20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef" Nov 04 01:28:41.620465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.620448 4312 scope.go:122] "RemoveContainer" containerID="b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9" Nov 04 01:28:41.630412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.630388 4312 scope.go:122] "RemoveContainer" containerID="20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef" Nov 04 01:28:41.630675 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:41.630658 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef\": container with ID starting with 20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef not found: ID does not exist" containerID="20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef" Nov 04 01:28:41.630759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.630683 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef"} err="failed to get container status \"20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef\": rpc error: code = NotFound desc = could not find container \"20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef\": container with ID starting with 20e8edb3a01045b29033fcdd0e6f5032a86b5bfff1bfb1e858f6669ab4c18eef not found: ID does not exist" Nov 04 01:28:41.630759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.630699 4312 scope.go:122] "RemoveContainer" containerID="b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9" Nov 04 01:28:41.630951 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:41.630932 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9\": container with ID starting with b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9 not found: ID does not exist" containerID="b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9" Nov 04 01:28:41.631010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.630955 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9"} err="failed to get container status \"b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9\": rpc error: code = NotFound desc = could not find container \"b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9\": container with ID starting with b9907f5f84bb19c9b865d73085dbfc8caa29a6aba7a735ae84125697d5a229a9 not found: ID does not exist" Nov 04 01:28:41.634261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.634238 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9"] Nov 04 01:28:41.638104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:41.638078 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-26f3d2-openshift-default-5b46f6689f-2t4c9"] Nov 04 01:28:42.110778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:42.110749 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" path="/var/lib/kubelet/pods/593b2775-f312-4b74-9c40-c0ab0fdf5b7d/volumes" Nov 04 01:28:42.848770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:42.848733 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw"] Nov 04 01:28:42.864298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:42.864271 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q"] Nov 04 01:28:44.066943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.066907 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" podUID="80adeed0-1499-4cfc-a002-114ae3bf45b7" containerName="payload-processing" containerID="cri-o://863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd" gracePeriod=25 Nov 04 01:28:44.076846 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.076804 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" podUID="070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" containerName="payload-pre-processing" containerID="cri-o://5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2" gracePeriod=25 Nov 04 01:28:44.351383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.351352 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:28:44.440170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.440135 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80adeed0-1499-4cfc-a002-114ae3bf45b7-kube-api-access-lzm2r\" (UniqueName: \"kubernetes.io/projected/80adeed0-1499-4cfc-a002-114ae3bf45b7-kube-api-access-lzm2r\") pod \"80adeed0-1499-4cfc-a002-114ae3bf45b7\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " Nov 04 01:28:44.440454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.440432 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80adeed0-1499-4cfc-a002-114ae3bf45b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80adeed0-1499-4cfc-a002-114ae3bf45b7-plugins-config-volume\") pod \"80adeed0-1499-4cfc-a002-114ae3bf45b7\" (UID: \"80adeed0-1499-4cfc-a002-114ae3bf45b7\") " Nov 04 01:28:44.441144 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.441103 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80adeed0-1499-4cfc-a002-114ae3bf45b7-plugins-config-volume" pod "80adeed0-1499-4cfc-a002-114ae3bf45b7" (UID: "80adeed0-1499-4cfc-a002-114ae3bf45b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.442988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.442966 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80adeed0-1499-4cfc-a002-114ae3bf45b7-kube-api-access-lzm2r" pod "80adeed0-1499-4cfc-a002-114ae3bf45b7" (UID: "80adeed0-1499-4cfc-a002-114ae3bf45b7"). InnerVolumeSpecName "kube-api-access-lzm2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.443405 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.443391 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:28:44.542438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.542407 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-plugins-config-volume\") pod \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " Nov 04 01:28:44.542599 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.542449 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-kube-api-access-kf2gl\" (UniqueName: \"kubernetes.io/projected/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-kube-api-access-kf2gl\") pod \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\" (UID: \"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a\") " Nov 04 01:28:44.542845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.542819 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-plugins-config-volume" pod "070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" (UID: "070e48e2-c5d8-4e5a-9c38-ae2ba902b75a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.543119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.543103 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.543167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.543123 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80adeed0-1499-4cfc-a002-114ae3bf45b7-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.543167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.543134 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzm2r\" (UniqueName: \"kubernetes.io/projected/80adeed0-1499-4cfc-a002-114ae3bf45b7-kube-api-access-lzm2r\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.544590 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.544568 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-kube-api-access-kf2gl" pod "070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" (UID: "070e48e2-c5d8-4e5a-9c38-ae2ba902b75a"). InnerVolumeSpecName "kube-api-access-kf2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.634299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.634228 4312 generic.go:356] "Generic (PLEG): container finished" podID="070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" containerID="5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2" exitCode=0 Nov 04 01:28:44.634299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.634275 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" Nov 04 01:28:44.634299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.634297 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" event={"ID":"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a","Type":"ContainerDied","Data":"5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2"} Nov 04 01:28:44.634589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.634359 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv" event={"ID":"070e48e2-c5d8-4e5a-9c38-ae2ba902b75a","Type":"ContainerDied","Data":"5d52f0ee5359ad20010ec6805cf5ce86d1759340f750fa46f952ac02a4eb9d76"} Nov 04 01:28:44.634589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.634384 4312 scope.go:122] "RemoveContainer" containerID="5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2" Nov 04 01:28:44.635804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.635782 4312 generic.go:356] "Generic (PLEG): container finished" podID="80adeed0-1499-4cfc-a002-114ae3bf45b7" containerID="863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd" exitCode=0 Nov 04 01:28:44.635914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.635848 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" event={"ID":"80adeed0-1499-4cfc-a002-114ae3bf45b7","Type":"ContainerDied","Data":"863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd"} Nov 04 01:28:44.635914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.635883 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" Nov 04 01:28:44.635914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.635888 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64" event={"ID":"80adeed0-1499-4cfc-a002-114ae3bf45b7","Type":"ContainerDied","Data":"a41269652a967a22e2a7612b0710af1929c716db28cfce624be592b3e8224ab3"} Nov 04 01:28:44.644240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.644218 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2gl\" (UniqueName: \"kubernetes.io/projected/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a-kube-api-access-kf2gl\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.646936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.646887 4312 scope.go:122] "RemoveContainer" containerID="5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2" Nov 04 01:28:44.647215 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:44.647191 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2\": container with ID starting with 5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2 not found: ID does not exist" containerID="5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2" Nov 04 01:28:44.647290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.647221 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2"} err="failed to get container status \"5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2\": rpc error: code = NotFound desc = could not find container \"5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2\": container with ID starting with 5cbb6a71c2c4118c8d5cab425918e2304a61155b2b10b6045befad7c3e27c6c2 not found: ID does not exist" Nov 04 01:28:44.647290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.647237 4312 scope.go:122] "RemoveContainer" containerID="863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd" Nov 04 01:28:44.656317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.656300 4312 scope.go:122] "RemoveContainer" containerID="863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd" Nov 04 01:28:44.656568 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:44.656548 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd\": container with ID starting with 863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd not found: ID does not exist" containerID="863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd" Nov 04 01:28:44.656620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.656574 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd"} err="failed to get container status \"863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd\": rpc error: code = NotFound desc = could not find container \"863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd\": container with ID starting with 863e7cd072d3f101a078c796d397c1b988608a3f08a9c63b57ef44bb21fc20bd not found: ID does not exist" Nov 04 01:28:44.664403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.663762 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv"] Nov 04 01:28:44.667523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.667505 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695ce9-7785748747-n7wqv"] Nov 04 01:28:44.678123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.678105 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64"] Nov 04 01:28:44.682730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:44.682712 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695ce9-74f76cc57b-l8w64"] Nov 04 01:28:45.162707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.162671 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl"] Nov 04 01:28:45.163197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.162899 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" podUID="e9b43751-6a7d-492a-9b6c-5d631d97abe6" containerName="maas-api" containerID="cri-o://afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411" gracePeriod=30 Nov 04 01:28:45.285284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.285257 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h"] Nov 04 01:28:45.285473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.285452 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" podUID="bb6f17fc-4a29-4637-92ea-d2a522dee9b3" containerName="maas-api" containerID="cri-o://b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf" gracePeriod=30 Nov 04 01:28:45.454367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.453592 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq"] Nov 04 01:28:45.454499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.454396 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" podUID="fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" containerName="maas-api" containerID="cri-o://b20f81b4bbc912f91b78662ef70160082d7aa6640944bedcf9e840037d9a6587" gracePeriod=30 Nov 04 01:28:45.461834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.461805 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" podUID="bb6f17fc-4a29-4637-92ea-d2a522dee9b3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.147:8443/health\": dial tcp 10.133.0.147:8443: connect: connection refused" Nov 04 01:28:45.465964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.465942 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:28:45.564839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.564804 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b43751-6a7d-492a-9b6c-5d631d97abe6-kube-api-access-f2q8x\" (UniqueName: \"kubernetes.io/projected/e9b43751-6a7d-492a-9b6c-5d631d97abe6-kube-api-access-f2q8x\") pod \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " Nov 04 01:28:45.564989 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.564947 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b43751-6a7d-492a-9b6c-5d631d97abe6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b43751-6a7d-492a-9b6c-5d631d97abe6-openshift-service-ca\") pod \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " Nov 04 01:28:45.565058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.565000 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls\") pod \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " Nov 04 01:28:45.565157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.565141 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-metrics-certs\") pod \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\" (UID: \"e9b43751-6a7d-492a-9b6c-5d631d97abe6\") " Nov 04 01:28:45.565514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.565285 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b43751-6a7d-492a-9b6c-5d631d97abe6-openshift-service-ca" pod "e9b43751-6a7d-492a-9b6c-5d631d97abe6" (UID: "e9b43751-6a7d-492a-9b6c-5d631d97abe6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:45.565771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.565751 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b43751-6a7d-492a-9b6c-5d631d97abe6-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.567031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.567006 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b43751-6a7d-492a-9b6c-5d631d97abe6-kube-api-access-f2q8x" pod "e9b43751-6a7d-492a-9b6c-5d631d97abe6" (UID: "e9b43751-6a7d-492a-9b6c-5d631d97abe6"). InnerVolumeSpecName "kube-api-access-f2q8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:45.567767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.567743 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-metrics-certs" pod "e9b43751-6a7d-492a-9b6c-5d631d97abe6" (UID: "e9b43751-6a7d-492a-9b6c-5d631d97abe6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:45.567876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.567795 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls" pod "e9b43751-6a7d-492a-9b6c-5d631d97abe6" (UID: "e9b43751-6a7d-492a-9b6c-5d631d97abe6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:45.574620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.574591 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:28:45.651838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.651033 4312 generic.go:356] "Generic (PLEG): container finished" podID="bb6f17fc-4a29-4637-92ea-d2a522dee9b3" containerID="b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf" exitCode=0 Nov 04 01:28:45.651838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.651168 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" event={"ID":"bb6f17fc-4a29-4637-92ea-d2a522dee9b3","Type":"ContainerDied","Data":"b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf"} Nov 04 01:28:45.651838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.651215 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" event={"ID":"bb6f17fc-4a29-4637-92ea-d2a522dee9b3","Type":"ContainerDied","Data":"79eb474692e90118e94d8338bab6628cafa498d6072c1134758726a4caa1577a"} Nov 04 01:28:45.651838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.651241 4312 scope.go:122] "RemoveContainer" containerID="b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf" Nov 04 01:28:45.651838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.651376 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h" Nov 04 01:28:45.654142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.654115 4312 generic.go:356] "Generic (PLEG): container finished" podID="fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" containerID="b20f81b4bbc912f91b78662ef70160082d7aa6640944bedcf9e840037d9a6587" exitCode=0 Nov 04 01:28:45.654232 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.654210 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" event={"ID":"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f","Type":"ContainerDied","Data":"b20f81b4bbc912f91b78662ef70160082d7aa6640944bedcf9e840037d9a6587"} Nov 04 01:28:45.658567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.658543 4312 generic.go:356] "Generic (PLEG): container finished" podID="e9b43751-6a7d-492a-9b6c-5d631d97abe6" containerID="afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411" exitCode=0 Nov 04 01:28:45.658682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.658604 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" Nov 04 01:28:45.658682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.658637 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" event={"ID":"e9b43751-6a7d-492a-9b6c-5d631d97abe6","Type":"ContainerDied","Data":"afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411"} Nov 04 01:28:45.658775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.658680 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl" event={"ID":"e9b43751-6a7d-492a-9b6c-5d631d97abe6","Type":"ContainerDied","Data":"3191c5829fcb98d1b5fc4a47061516ee9075559af1ce3410a007aa5354916ad8"} Nov 04 01:28:45.666387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.666366 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-openshift-service-ca\") pod \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " Nov 04 01:28:45.666618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.666603 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-maas-api-tls\") pod \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " Nov 04 01:28:45.667005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.666746 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-kube-api-access-mmcsk\" (UniqueName: \"kubernetes.io/projected/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-kube-api-access-mmcsk\") pod \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " Nov 04 01:28:45.667005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.666757 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-openshift-service-ca" pod "bb6f17fc-4a29-4637-92ea-d2a522dee9b3" (UID: "bb6f17fc-4a29-4637-92ea-d2a522dee9b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:45.667005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.666782 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-metrics-certs\") pod \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\" (UID: \"bb6f17fc-4a29-4637-92ea-d2a522dee9b3\") " Nov 04 01:28:45.667554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.667322 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.667554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.667380 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2q8x\" (UniqueName: \"kubernetes.io/projected/e9b43751-6a7d-492a-9b6c-5d631d97abe6-kube-api-access-f2q8x\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.667554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.667397 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.667554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.667414 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b43751-6a7d-492a-9b6c-5d631d97abe6-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.667981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.667962 4312 scope.go:122] "RemoveContainer" containerID="b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf" Nov 04 01:28:45.668632 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:45.668605 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf\": container with ID starting with b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf not found: ID does not exist" containerID="b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf" Nov 04 01:28:45.668737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.668642 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf"} err="failed to get container status \"b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf\": rpc error: code = NotFound desc = could not find container \"b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf\": container with ID starting with b31382c270a91442a78fbc35fbd19fe927a30410154e0e4f29f80e552e886bcf not found: ID does not exist" Nov 04 01:28:45.668737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.668667 4312 scope.go:122] "RemoveContainer" containerID="afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411" Nov 04 01:28:45.670698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.670678 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-metrics-certs" pod "bb6f17fc-4a29-4637-92ea-d2a522dee9b3" (UID: "bb6f17fc-4a29-4637-92ea-d2a522dee9b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:45.671269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.671176 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-kube-api-access-mmcsk" pod "bb6f17fc-4a29-4637-92ea-d2a522dee9b3" (UID: "bb6f17fc-4a29-4637-92ea-d2a522dee9b3"). InnerVolumeSpecName "kube-api-access-mmcsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:45.672174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.672148 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-maas-api-tls" pod "bb6f17fc-4a29-4637-92ea-d2a522dee9b3" (UID: "bb6f17fc-4a29-4637-92ea-d2a522dee9b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:45.684462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.684416 4312 scope.go:122] "RemoveContainer" containerID="afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411" Nov 04 01:28:45.684904 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:45.684812 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411\": container with ID starting with afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411 not found: ID does not exist" containerID="afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411" Nov 04 01:28:45.684904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.684852 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411"} err="failed to get container status \"afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411\": rpc error: code = NotFound desc = could not find container \"afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411\": container with ID starting with afc8e5bafff14cf706f4b12e7ae6a4605dec25f2daf198c6244d189531114411 not found: ID does not exist" Nov 04 01:28:45.692849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.692825 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl"] Nov 04 01:28:45.697280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.697260 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0c8e8-86576d786b-ghrvl"] Nov 04 01:28:45.769176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.769138 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmcsk\" (UniqueName: \"kubernetes.io/projected/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-kube-api-access-mmcsk\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.769398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.769382 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.769504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.769492 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6f17fc-4a29-4637-92ea-d2a522dee9b3-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.815966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.815639 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:28:45.971999 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.971968 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-maas-api-tls\") pod \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " Nov 04 01:28:45.972170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.972152 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-metrics-certs\") pod \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " Nov 04 01:28:45.972237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.972220 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-openshift-service-ca\") pod \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " Nov 04 01:28:45.972379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.972364 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-kube-api-access-5hdk5\" (UniqueName: \"kubernetes.io/projected/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-kube-api-access-5hdk5\") pod \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\" (UID: \"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f\") " Nov 04 01:28:45.973276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.973237 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-openshift-service-ca" pod "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" (UID: "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:45.974123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.974100 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-metrics-certs" pod "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" (UID: "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:45.974386 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.974362 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-maas-api-tls" pod "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" (UID: "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:45.974512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:45.974490 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-kube-api-access-5hdk5" pod "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" (UID: "fb100544-48c1-4288-b4c0-a8f3ae9c4c2f"). InnerVolumeSpecName "kube-api-access-5hdk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:46.009359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.004421 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h"] Nov 04 01:28:46.009359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.009030 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-695ce9-7d5bc9cdcb-v759h"] Nov 04 01:28:46.074475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.074446 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdk5\" (UniqueName: \"kubernetes.io/projected/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-kube-api-access-5hdk5\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:46.074475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.074473 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:46.074475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.074483 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:46.074712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.074495 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:46.109097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.109068 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" path="/var/lib/kubelet/pods/070e48e2-c5d8-4e5a-9c38-ae2ba902b75a/volumes" Nov 04 01:28:46.109565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.109551 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80adeed0-1499-4cfc-a002-114ae3bf45b7" path="/var/lib/kubelet/pods/80adeed0-1499-4cfc-a002-114ae3bf45b7/volumes" Nov 04 01:28:46.109942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.109929 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6f17fc-4a29-4637-92ea-d2a522dee9b3" path="/var/lib/kubelet/pods/bb6f17fc-4a29-4637-92ea-d2a522dee9b3/volumes" Nov 04 01:28:46.110444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.110424 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b43751-6a7d-492a-9b6c-5d631d97abe6" path="/var/lib/kubelet/pods/e9b43751-6a7d-492a-9b6c-5d631d97abe6/volumes" Nov 04 01:28:46.668111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.668086 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" Nov 04 01:28:46.668595 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.668110 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq" event={"ID":"fb100544-48c1-4288-b4c0-a8f3ae9c4c2f","Type":"ContainerDied","Data":"ceb4bc5659bfdc537f6c649fa1f89755fb099438771f1e27eebec25f5394e63f"} Nov 04 01:28:46.668595 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.668171 4312 scope.go:122] "RemoveContainer" containerID="b20f81b4bbc912f91b78662ef70160082d7aa6640944bedcf9e840037d9a6587" Nov 04 01:28:46.688033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.688007 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq"] Nov 04 01:28:46.692376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:46.692355 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ba844c-78bc6f7ff-ldnzq"] Nov 04 01:28:47.849555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:47.849515 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" podUID="ff1315cd-6359-47c3-b4c1-7db9e1480078" containerName="payload-processing" containerID="cri-o://a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a" gracePeriod=25 Nov 04 01:28:47.865011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:47.864960 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" podUID="7e1ec55d-d260-4d44-a5fb-3401330091f3" containerName="payload-pre-processing" containerID="cri-o://959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0" gracePeriod=25 Nov 04 01:28:48.125827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.125796 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" path="/var/lib/kubelet/pods/fb100544-48c1-4288-b4c0-a8f3ae9c4c2f/volumes" Nov 04 01:28:48.141422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.141401 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:28:48.193575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.193549 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:28:48.313295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.313261 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1ec55d-d260-4d44-a5fb-3401330091f3-kube-api-access-zhnpg\" (UniqueName: \"kubernetes.io/projected/7e1ec55d-d260-4d44-a5fb-3401330091f3-kube-api-access-zhnpg\") pod \"7e1ec55d-d260-4d44-a5fb-3401330091f3\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " Nov 04 01:28:48.313489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.313395 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1315cd-6359-47c3-b4c1-7db9e1480078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1315cd-6359-47c3-b4c1-7db9e1480078-plugins-config-volume\") pod \"ff1315cd-6359-47c3-b4c1-7db9e1480078\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " Nov 04 01:28:48.313489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.313442 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1ec55d-d260-4d44-a5fb-3401330091f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ec55d-d260-4d44-a5fb-3401330091f3-plugins-config-volume\") pod \"7e1ec55d-d260-4d44-a5fb-3401330091f3\" (UID: \"7e1ec55d-d260-4d44-a5fb-3401330091f3\") " Nov 04 01:28:48.313606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.313582 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1315cd-6359-47c3-b4c1-7db9e1480078-kube-api-access-dlxcb\" (UniqueName: \"kubernetes.io/projected/ff1315cd-6359-47c3-b4c1-7db9e1480078-kube-api-access-dlxcb\") pod \"ff1315cd-6359-47c3-b4c1-7db9e1480078\" (UID: \"ff1315cd-6359-47c3-b4c1-7db9e1480078\") " Nov 04 01:28:48.313802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.313778 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1315cd-6359-47c3-b4c1-7db9e1480078-plugins-config-volume" pod "ff1315cd-6359-47c3-b4c1-7db9e1480078" (UID: "ff1315cd-6359-47c3-b4c1-7db9e1480078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:48.313877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.313845 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1ec55d-d260-4d44-a5fb-3401330091f3-plugins-config-volume" pod "7e1ec55d-d260-4d44-a5fb-3401330091f3" (UID: "7e1ec55d-d260-4d44-a5fb-3401330091f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:48.314098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.314083 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1315cd-6359-47c3-b4c1-7db9e1480078-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:48.314141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.314101 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ec55d-d260-4d44-a5fb-3401330091f3-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:48.315407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.315390 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1ec55d-d260-4d44-a5fb-3401330091f3-kube-api-access-zhnpg" pod "7e1ec55d-d260-4d44-a5fb-3401330091f3" (UID: "7e1ec55d-d260-4d44-a5fb-3401330091f3"). InnerVolumeSpecName "kube-api-access-zhnpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:48.315493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.315471 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1315cd-6359-47c3-b4c1-7db9e1480078-kube-api-access-dlxcb" pod "ff1315cd-6359-47c3-b4c1-7db9e1480078" (UID: "ff1315cd-6359-47c3-b4c1-7db9e1480078"). InnerVolumeSpecName "kube-api-access-dlxcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:48.414981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.414894 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxcb\" (UniqueName: \"kubernetes.io/projected/ff1315cd-6359-47c3-b4c1-7db9e1480078-kube-api-access-dlxcb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:48.414981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.414925 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhnpg\" (UniqueName: \"kubernetes.io/projected/7e1ec55d-d260-4d44-a5fb-3401330091f3-kube-api-access-zhnpg\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:48.684897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.684805 4312 generic.go:356] "Generic (PLEG): container finished" podID="7e1ec55d-d260-4d44-a5fb-3401330091f3" containerID="959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0" exitCode=0 Nov 04 01:28:48.684897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.684869 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" Nov 04 01:28:48.685116 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.684889 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" event={"ID":"7e1ec55d-d260-4d44-a5fb-3401330091f3","Type":"ContainerDied","Data":"959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0"} Nov 04 01:28:48.685116 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.684973 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q" event={"ID":"7e1ec55d-d260-4d44-a5fb-3401330091f3","Type":"ContainerDied","Data":"00a5de945db5dc5dbd7870b048b50b8fbccf66573377a12263541693930101ab"} Nov 04 01:28:48.685116 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.685011 4312 scope.go:122] "RemoveContainer" containerID="959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0" Nov 04 01:28:48.686403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.686379 4312 generic.go:356] "Generic (PLEG): container finished" podID="ff1315cd-6359-47c3-b4c1-7db9e1480078" containerID="a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a" exitCode=0 Nov 04 01:28:48.686499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.686436 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" event={"ID":"ff1315cd-6359-47c3-b4c1-7db9e1480078","Type":"ContainerDied","Data":"a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a"} Nov 04 01:28:48.686499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.686448 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" Nov 04 01:28:48.686499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.686477 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw" event={"ID":"ff1315cd-6359-47c3-b4c1-7db9e1480078","Type":"ContainerDied","Data":"90beebb78609a40f9430a4d8ef73af0dab80284d858b1d0a83289aee59775c6f"} Nov 04 01:28:48.697963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.697945 4312 scope.go:122] "RemoveContainer" containerID="959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0" Nov 04 01:28:48.698254 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:48.698236 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0\": container with ID starting with 959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0 not found: ID does not exist" containerID="959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0" Nov 04 01:28:48.698299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.698264 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0"} err="failed to get container status \"959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0\": rpc error: code = NotFound desc = could not find container \"959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0\": container with ID starting with 959c1569f9feec3714050d9beb1b9591a79b763dc37d21e5f964329f1c95b3b0 not found: ID does not exist" Nov 04 01:28:48.698299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.698288 4312 scope.go:122] "RemoveContainer" containerID="a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a" Nov 04 01:28:48.707971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.707956 4312 scope.go:122] "RemoveContainer" containerID="a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a" Nov 04 01:28:48.708208 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:48.708189 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a\": container with ID starting with a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a not found: ID does not exist" containerID="a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a" Nov 04 01:28:48.708265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.708218 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a"} err="failed to get container status \"a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a\": rpc error: code = NotFound desc = could not find container \"a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a\": container with ID starting with a0e80c77ac5e37880dc0639d9249fa7f6fe5ff717f8cef57b30c45bf13d73c7a not found: ID does not exist" Nov 04 01:28:48.713771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.713749 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q"] Nov 04 01:28:48.718418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.718399 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0c8e8-54c746697f-5h82q"] Nov 04 01:28:48.729610 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.729590 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw"] Nov 04 01:28:48.740996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:48.740978 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0c8e8-55b7d8f58d-cnwjw"] Nov 04 01:28:50.109147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:50.109121 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1ec55d-d260-4d44-a5fb-3401330091f3" path="/var/lib/kubelet/pods/7e1ec55d-d260-4d44-a5fb-3401330091f3/volumes" Nov 04 01:28:50.109676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:50.109657 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1315cd-6359-47c3-b4c1-7db9e1480078" path="/var/lib/kubelet/pods/ff1315cd-6359-47c3-b4c1-7db9e1480078/volumes" Nov 04 01:28:59.034458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:59.034370 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24"] Nov 04 01:28:59.034954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:28:59.034637 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" containerID="cri-o://a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852" gracePeriod=30 Nov 04 01:28:59.458726 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:28:59.458637 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:29:00.252075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.247791 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5"] Nov 04 01:29:00.252075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.247995 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" containerID="cri-o://055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e" gracePeriod=30 Nov 04 01:29:00.376367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.366799 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf"] Nov 04 01:29:00.376367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.367099 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" containerID="cri-o://5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f" gracePeriod=30 Nov 04 01:29:00.459474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.459438 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr"] Nov 04 01:29:00.460072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460055 4312 cpu_manager.go:414] "containerMap: removing container" podUID="64548438-f3dd-488f-a623-9cd17e97af8e" containerName="maas-api" Nov 04 01:29:00.460072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460070 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64548438-f3dd-488f-a623-9cd17e97af8e" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460081 4312 cpu_manager.go:414] "containerMap: removing container" podUID="bb6f17fc-4a29-4637-92ea-d2a522dee9b3" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460086 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6f17fc-4a29-4637-92ea-d2a522dee9b3" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460096 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460102 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460113 4312 cpu_manager.go:414] "containerMap: removing container" podUID="03d9433c-03e4-4a44-889f-bed3f3c8a24c" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460118 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d9433c-03e4-4a44-889f-bed3f3c8a24c" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460130 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ff1315cd-6359-47c3-b4c1-7db9e1480078" containerName="payload-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460135 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1315cd-6359-47c3-b4c1-7db9e1480078" containerName="payload-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460141 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460146 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460162 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2b81fcbb-dfaf-467c-9da0-913c1f77a907" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460171 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b81fcbb-dfaf-467c-9da0-913c1f77a907" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460186 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" containerName="payload-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460191 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" containerName="payload-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460199 4312 cpu_manager.go:414] "containerMap: removing container" podUID="51241f78-295d-4d43-8ed0-dab0ba0e7503" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460204 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51241f78-295d-4d43-8ed0-dab0ba0e7503" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460217 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460221 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460236 4312 cpu_manager.go:414] "containerMap: removing container" podUID="070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460240 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460250 4312 cpu_manager.go:414] "containerMap: removing container" podUID="80adeed0-1499-4cfc-a002-114ae3bf45b7" containerName="payload-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460255 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80adeed0-1499-4cfc-a002-114ae3bf45b7" containerName="payload-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460262 4312 cpu_manager.go:414] "containerMap: removing container" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460267 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460274 4312 cpu_manager.go:414] "containerMap: removing container" podUID="7e1ec55d-d260-4d44-a5fb-3401330091f3" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460279 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1ec55d-d260-4d44-a5fb-3401330091f3" containerName="payload-pre-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460286 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e9b43751-6a7d-492a-9b6c-5d631d97abe6" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460291 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b43751-6a7d-492a-9b6c-5d631d97abe6" containerName="maas-api" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460304 4312 cpu_manager.go:414] "containerMap: removing container" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460308 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460318 4312 cpu_manager.go:414] "containerMap: removing container" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460322 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460344 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6d423e64-8112-41ed-a035-5f15b34cc2fe" containerName="payload-processing" Nov 04 01:29:00.460306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460351 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d423e64-8112-41ed-a035-5f15b34cc2fe" containerName="payload-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460486 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="64548438-f3dd-488f-a623-9cd17e97af8e" containerName="maas-api" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460501 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6f17fc-4a29-4637-92ea-d2a522dee9b3" containerName="maas-api" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460512 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5357bfb-550e-4296-80f1-939d795a545e" containerName="istio-proxy" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460522 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d9433c-03e4-4a44-889f-bed3f3c8a24c" containerName="payload-pre-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460532 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1315cd-6359-47c3-b4c1-7db9e1480078" containerName="payload-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460545 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5ed29b-ec80-4bef-bc71-4c185119193a" containerName="istio-proxy" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460558 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b81fcbb-dfaf-467c-9da0-913c1f77a907" containerName="maas-api" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460567 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2138bbdb-3ef7-4ad4-83d5-ac22fabcd759" containerName="payload-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460575 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="51241f78-295d-4d43-8ed0-dab0ba0e7503" containerName="payload-pre-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460585 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb100544-48c1-4288-b4c0-a8f3ae9c4c2f" containerName="maas-api" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460602 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="070e48e2-c5d8-4e5a-9c38-ae2ba902b75a" containerName="payload-pre-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460613 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="80adeed0-1499-4cfc-a002-114ae3bf45b7" containerName="payload-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460621 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460628 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1ec55d-d260-4d44-a5fb-3401330091f3" containerName="payload-pre-processing" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460638 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b43751-6a7d-492a-9b6c-5d631d97abe6" containerName="maas-api" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460650 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b6ab23-7925-4dc8-a773-f70db1a2fe06" containerName="istio-proxy" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460658 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="593b2775-f312-4b74-9c40-c0ab0fdf5b7d" containerName="istio-proxy" Nov 04 01:29:00.462073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.460663 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d423e64-8112-41ed-a035-5f15b34cc2fe" containerName="payload-processing" Nov 04 01:29:00.466662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.466643 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.478804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.478779 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr"] Nov 04 01:29:00.563540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.563499 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a362b6-openshift-default-dockercfg-7zkzs\"" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.570542 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkfdx\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-kube-api-access-nkfdx\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.570676 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-token\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.570709 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-podinfo\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.570747 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-data\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.570802 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.570942 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-envoy\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.571005 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-credential-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.571032 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4204c8e6-9476-4a48-a947-635f16cb21c7-istiod-ca-cert\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.577635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.571065 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-certs\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672476 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkfdx\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-kube-api-access-nkfdx\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672545 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-token\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672572 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-podinfo\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672605 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-data\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672655 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672774 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-envoy\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672917 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-credential-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.672963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672958 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4204c8e6-9476-4a48-a947-635f16cb21c7-istiod-ca-cert\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.673066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.672993 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-certs\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.673138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.673114 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-data\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.673200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.673147 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.673279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.673259 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-credential-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.673444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.673428 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-certs\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.673596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.673578 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4204c8e6-9476-4a48-a947-635f16cb21c7-istiod-ca-cert\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.675749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.675727 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-envoy\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.675905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.675889 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-podinfo\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.688907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.688885 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkfdx\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-kube-api-access-nkfdx\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.689013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.688942 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-token\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:00.782143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:00.782115 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:01.081680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.081647 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr"] Nov 04 01:29:01.082165 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:01.082142 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4204c8e6_9476_4a48_a947_635f16cb21c7.slice/crio-e71407aa36090cf168276ec90417dcb9ff4b01e42d429e091078a97c782d2b26 WatchSource:0}: Error finding container e71407aa36090cf168276ec90417dcb9ff4b01e42d429e091078a97c782d2b26: Status 404 returned error can't find the container with id e71407aa36090cf168276ec90417dcb9ff4b01e42d429e091078a97c782d2b26 Nov 04 01:29:01.088022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.085054 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:01.088022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.085220 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:01.088022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.085312 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:01.777257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.777228 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" event={"ID":"4204c8e6-9476-4a48-a947-635f16cb21c7","Type":"ContainerStarted","Data":"4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191"} Nov 04 01:29:01.777741 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.777307 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" event={"ID":"4204c8e6-9476-4a48-a947-635f16cb21c7","Type":"ContainerStarted","Data":"e71407aa36090cf168276ec90417dcb9ff4b01e42d429e091078a97c782d2b26"} Nov 04 01:29:01.782554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.782532 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:01.804094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:01.804046 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" podStartSLOduration=1.80403384 podStartE2EDuration="1.80403384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:01.801442177 +0000 UTC m=+1573.824374210" watchObservedRunningTime="2026-11-04 01:29:01.80403384 +0000 UTC m=+1573.826965850" Nov 04 01:29:02.788934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:02.788899 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:03.792173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:03.792142 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:03.793046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:03.793028 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:03.859688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:03.859649 4312 patch_prober.go:29] interesting pod/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:29:03.859855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:03.859716 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:29:04.377305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.377282 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:29:04.421980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.421947 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-envoy\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422038 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-credential-socket\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422095 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-certs\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422273 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-kube-api-access-54w4d\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-kube-api-access-54w4d\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422326 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istiod-ca-cert\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422394 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-token\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-token\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422322 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-credential-socket" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422402 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-certs" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422424 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-podinfo\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422514 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-data\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422587 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-socket\") pod \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\" (UID: \"3379949f-d9f1-4c2e-9341-407cbdfc7b85\") " Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422754 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istiod-ca-cert" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.422979 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-data" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.423065 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-socket" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.423307 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.423328 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.423358 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.423374 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.423387 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.424636 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-podinfo" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.425278 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-envoy" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:04.425427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.425393 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-token" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:04.426817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.426788 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-kube-api-access-54w4d" pod "3379949f-d9f1-4c2e-9341-407cbdfc7b85" (UID: "3379949f-d9f1-4c2e-9341-407cbdfc7b85"). InnerVolumeSpecName "kube-api-access-54w4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:04.524281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.524243 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.524281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.524271 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54w4d\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-kube-api-access-54w4d\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.524499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.524289 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.524499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.524303 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379949f-d9f1-4c2e-9341-407cbdfc7b85-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:04.800998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.800963 4312 generic.go:356] "Generic (PLEG): container finished" podID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerID="a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852" exitCode=0 Nov 04 01:29:04.801513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.801014 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" Nov 04 01:29:04.801513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.801042 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" event={"ID":"3379949f-d9f1-4c2e-9341-407cbdfc7b85","Type":"ContainerDied","Data":"a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852"} Nov 04 01:29:04.801513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.801108 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24" event={"ID":"3379949f-d9f1-4c2e-9341-407cbdfc7b85","Type":"ContainerDied","Data":"ad17f318b8394d53e17fb94099f7a42c2ae267ba0f7c05d38bf03f9fe738e796"} Nov 04 01:29:04.801513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.801132 4312 scope.go:122] "RemoveContainer" containerID="a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852" Nov 04 01:29:04.812316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.812302 4312 scope.go:122] "RemoveContainer" containerID="af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa" Nov 04 01:29:04.821175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.821159 4312 scope.go:122] "RemoveContainer" containerID="a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852" Nov 04 01:29:04.821488 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:04.821466 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852\": container with ID starting with a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852 not found: ID does not exist" containerID="a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852" Nov 04 01:29:04.821533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.821496 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852"} err="failed to get container status \"a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852\": rpc error: code = NotFound desc = could not find container \"a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852\": container with ID starting with a0c530d920a395c7e92ec40aade3c79577221eb1d625dd190bff5e189d005852 not found: ID does not exist" Nov 04 01:29:04.821533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.821511 4312 scope.go:122] "RemoveContainer" containerID="af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa" Nov 04 01:29:04.821760 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:04.821743 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa\": container with ID starting with af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa not found: ID does not exist" containerID="af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa" Nov 04 01:29:04.821804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.821767 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa"} err="failed to get container status \"af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa\": rpc error: code = NotFound desc = could not find container \"af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa\": container with ID starting with af3503dd94882c2236cf37c6821e2ebb1cf6dcf1a80dec9137a15863b1856faa not found: ID does not exist" Nov 04 01:29:04.829024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.829002 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24"] Nov 04 01:29:04.833525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:04.833504 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d0c8e8-openshift-default-54d98c59-97k24"] Nov 04 01:29:05.405678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.405648 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq"] Nov 04 01:29:05.407032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.406714 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" Nov 04 01:29:05.407032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.406737 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" Nov 04 01:29:05.407032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.406956 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" Nov 04 01:29:05.407032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.406972 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" Nov 04 01:29:05.427312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.427284 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq"] Nov 04 01:29:05.427599 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.427572 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:05.435535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.435512 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a362b6-dockercfg-9hsx7\"" Nov 04 01:29:05.436724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.436703 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a362b6\"" Nov 04 01:29:05.441642 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.441618 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z"] Nov 04 01:29:05.442720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.442698 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" Nov 04 01:29:05.442842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.442830 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" containerName="istio-proxy" Nov 04 01:29:05.448880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.448849 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.456922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.456785 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z"] Nov 04 01:29:05.657648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.657613 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:29:05.673617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.672576 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8fp\" (UniqueName: \"kubernetes.io/projected/236a8c22-110b-4dd3-a3b0-3687494fd1dc-kube-api-access-sb8fp\") pod \"payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:05.673617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.672693 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mld\" (UniqueName: \"kubernetes.io/projected/8733aede-0f22-4ebe-90e5-636a7409a56d-kube-api-access-g2mld\") pod \"payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.673617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.672786 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8733aede-0f22-4ebe-90e5-636a7409a56d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.673617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.672842 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236a8c22-110b-4dd3-a3b0-3687494fd1dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:05.773691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.773662 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-podinfo\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.773867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.773709 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-token\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.773867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.773855 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-kube-api-access-vxf5j\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-kube-api-access-vxf5j\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.773994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.773943 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-data\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.774055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.773992 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-certs\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.774128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.774111 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-envoy\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.774196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.774162 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istiod-ca-cert\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.774256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.774200 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-credential-socket\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.774256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.774245 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-socket\") pod \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\" (UID: \"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910\") " Nov 04 01:29:05.774653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.774624 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8fp\" (UniqueName: \"kubernetes.io/projected/236a8c22-110b-4dd3-a3b0-3687494fd1dc-kube-api-access-sb8fp\") pod \"payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:05.774817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.774751 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mld\" (UniqueName: \"kubernetes.io/projected/8733aede-0f22-4ebe-90e5-636a7409a56d-kube-api-access-g2mld\") pod \"payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.775001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.774977 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8733aede-0f22-4ebe-90e5-636a7409a56d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.775092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.775056 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236a8c22-110b-4dd3-a3b0-3687494fd1dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:05.775155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.775129 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-data" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.775301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.775081 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-socket" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.775392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.775249 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istiod-ca-cert" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:05.775849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.775822 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-certs" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.776050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.776018 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8733aede-0f22-4ebe-90e5-636a7409a56d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.776136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.776081 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-credential-socket" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.776247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.776223 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236a8c22-110b-4dd3-a3b0-3687494fd1dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:05.776688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.776665 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-podinfo" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:29:05.776772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.776726 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-kube-api-access-vxf5j" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "kube-api-access-vxf5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:05.777257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.777234 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-token" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:05.777982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.777964 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-envoy" pod "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" (UID: "2cdd5980-cd0c-4f7c-9e15-4b2b6d828910"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.787143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.787121 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8fp\" (UniqueName: \"kubernetes.io/projected/236a8c22-110b-4dd3-a3b0-3687494fd1dc-kube-api-access-sb8fp\") pod \"payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:05.787244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.787205 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mld\" (UniqueName: \"kubernetes.io/projected/8733aede-0f22-4ebe-90e5-636a7409a56d-kube-api-access-g2mld\") pod \"payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.799995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.799949 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:29:05.809304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.809280 4312 generic.go:356] "Generic (PLEG): container finished" podID="abc63742-f448-4195-bbe2-d15647027916" containerID="5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f" exitCode=0 Nov 04 01:29:05.809661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.809321 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" Nov 04 01:29:05.809661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.809360 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerDied","Data":"5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f"} Nov 04 01:29:05.809661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.809408 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf" event={"ID":"abc63742-f448-4195-bbe2-d15647027916","Type":"ContainerDied","Data":"fd4a44635982220716ab7940abb0c690d6c6454a3e243a3a5559a4f02aa2090d"} Nov 04 01:29:05.809661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.809435 4312 scope.go:122] "RemoveContainer" containerID="5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f" Nov 04 01:29:05.811240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.811220 4312 generic.go:356] "Generic (PLEG): container finished" podID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerID="055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e" exitCode=0 Nov 04 01:29:05.811360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.811265 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" Nov 04 01:29:05.811360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.811303 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" event={"ID":"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910","Type":"ContainerDied","Data":"055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e"} Nov 04 01:29:05.811480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.811363 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5" event={"ID":"2cdd5980-cd0c-4f7c-9e15-4b2b6d828910","Type":"ContainerDied","Data":"e7e9722abeec9e9b83fc1a6e15e0210cfc3905caa5f786a1f54b6a73abd4ee6a"} Nov 04 01:29:05.819412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.819394 4312 scope.go:122] "RemoveContainer" containerID="fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1" Nov 04 01:29:05.829731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.829709 4312 scope.go:122] "RemoveContainer" containerID="5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f" Nov 04 01:29:05.830827 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:05.830795 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f\": container with ID starting with 5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f not found: ID does not exist" containerID="5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f" Nov 04 01:29:05.830943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.830836 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f"} err="failed to get container status \"5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f\": rpc error: code = NotFound desc = could not find container \"5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f\": container with ID starting with 5f49dc351ac4f5233893d085c592b0753a98b0fe160670bb1f0b818bd8e6f93f not found: ID does not exist" Nov 04 01:29:05.830943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.830859 4312 scope.go:122] "RemoveContainer" containerID="fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1" Nov 04 01:29:05.831271 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:05.831238 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1\": container with ID starting with fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1 not found: ID does not exist" containerID="fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1" Nov 04 01:29:05.831696 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.831272 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1"} err="failed to get container status \"fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1\": rpc error: code = NotFound desc = could not find container \"fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1\": container with ID starting with fa79b77da451df4a7de8f058b2e46ebf7e1ec68d488a003a575bb0fab1390ca1 not found: ID does not exist" Nov 04 01:29:05.831696 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.831293 4312 scope.go:122] "RemoveContainer" containerID="055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e" Nov 04 01:29:05.841674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.841650 4312 scope.go:122] "RemoveContainer" containerID="86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b" Nov 04 01:29:05.850130 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.850104 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5"] Nov 04 01:29:05.852675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.852657 4312 scope.go:122] "RemoveContainer" containerID="055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e" Nov 04 01:29:05.852973 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:05.852954 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e\": container with ID starting with 055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e not found: ID does not exist" containerID="055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e" Nov 04 01:29:05.853034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.852980 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e"} err="failed to get container status \"055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e\": rpc error: code = NotFound desc = could not find container \"055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e\": container with ID starting with 055831be6afef71a64953a27789965890527c4edbee11bde0381ed09c955d37e not found: ID does not exist" Nov 04 01:29:05.853034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.852996 4312 scope.go:122] "RemoveContainer" containerID="86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b" Nov 04 01:29:05.853279 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:05.853262 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b\": container with ID starting with 86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b not found: ID does not exist" containerID="86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b" Nov 04 01:29:05.853325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.853284 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b"} err="failed to get container status \"86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b\": rpc error: code = NotFound desc = could not find container \"86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b\": container with ID starting with 86f84b9a01bd292d7e00e85554e093e5230b55d23d105d05f0592faa4e98509b not found: ID does not exist" Nov 04 01:29:05.854573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.854548 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-695ce9-openshift-default-5b67ddf675-xmtv5"] Nov 04 01:29:05.876103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876077 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxf5j\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-kube-api-access-vxf5j\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876106 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876121 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876135 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876148 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876160 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876173 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876186 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.876223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.876199 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:05.904888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.904864 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:05.977380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977350 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-socket\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977452 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc63742-f448-4195-bbe2-d15647027916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc63742-f448-4195-bbe2-d15647027916-istiod-ca-cert\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977510 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-envoy\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977605 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977591 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-credential-socket\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977642 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-data\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977713 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-certs\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977759 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abc63742-f448-4195-bbe2-d15647027916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc63742-f448-4195-bbe2-d15647027916-istio-podinfo\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977807 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-kube-api-access-d7hhj\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-kube-api-access-d7hhj\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.977861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.977843 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-istio-token\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-istio-token\") pod \"abc63742-f448-4195-bbe2-d15647027916\" (UID: \"abc63742-f448-4195-bbe2-d15647027916\") " Nov 04 01:29:05.978857 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.978832 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-certs" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.979127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.979103 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-data" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.979249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.979221 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-socket" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.979468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.979403 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-credential-socket" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.979627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.979602 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc63742-f448-4195-bbe2-d15647027916-istiod-ca-cert" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:05.981065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.981045 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-istio-token" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:05.981182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.981154 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-envoy" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:05.981473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.981453 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-kube-api-access-d7hhj" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "kube-api-access-d7hhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:05.983147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:05.983123 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc63742-f448-4195-bbe2-d15647027916-istio-podinfo" pod "abc63742-f448-4195-bbe2-d15647027916" (UID: "abc63742-f448-4195-bbe2-d15647027916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:29:06.079519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079443 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079473 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc63742-f448-4195-bbe2-d15647027916-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079488 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079506 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079521 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079536 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc63742-f448-4195-bbe2-d15647027916-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079550 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc63742-f448-4195-bbe2-d15647027916-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079564 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hhj\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-kube-api-access-d7hhj\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.079789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.079578 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc63742-f448-4195-bbe2-d15647027916-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.084535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.084509 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:06.110086 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.110056 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" path="/var/lib/kubelet/pods/2cdd5980-cd0c-4f7c-9e15-4b2b6d828910/volumes" Nov 04 01:29:06.110712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.110692 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3379949f-d9f1-4c2e-9341-407cbdfc7b85" path="/var/lib/kubelet/pods/3379949f-d9f1-4c2e-9341-407cbdfc7b85/volumes" Nov 04 01:29:06.139012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.138982 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf"] Nov 04 01:29:06.144608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.144579 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d2e4e3-openshift-default-d49f8cb9f-bp7pf"] Nov 04 01:29:06.202705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.202670 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z"] Nov 04 01:29:06.204229 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:06.204204 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8733aede_0f22_4ebe_90e5_636a7409a56d.slice/crio-3aec2edf96bcd634344798234a0743c789b7d623dbe956b254c5ee72b5d46bef WatchSource:0}: Error finding container 3aec2edf96bcd634344798234a0743c789b7d623dbe956b254c5ee72b5d46bef: Status 404 returned error can't find the container with id 3aec2edf96bcd634344798234a0743c789b7d623dbe956b254c5ee72b5d46bef Nov 04 01:29:06.412301 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:06.412276 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236a8c22_110b_4dd3_a3b0_3687494fd1dc.slice/crio-7d1289cc3fa5f607385cd0c86ce21bc01609906b2c3401f53bf87f5d4a2f7bcd WatchSource:0}: Error finding container 7d1289cc3fa5f607385cd0c86ce21bc01609906b2c3401f53bf87f5d4a2f7bcd: Status 404 returned error can't find the container with id 7d1289cc3fa5f607385cd0c86ce21bc01609906b2c3401f53bf87f5d4a2f7bcd Nov 04 01:29:06.412829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.412805 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq"] Nov 04 01:29:06.820067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.820029 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" event={"ID":"8733aede-0f22-4ebe-90e5-636a7409a56d","Type":"ContainerStarted","Data":"41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf"} Nov 04 01:29:06.820504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.820097 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:06.820504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.820124 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" event={"ID":"8733aede-0f22-4ebe-90e5-636a7409a56d","Type":"ContainerStarted","Data":"3aec2edf96bcd634344798234a0743c789b7d623dbe956b254c5ee72b5d46bef"} Nov 04 01:29:06.821750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.821724 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" event={"ID":"236a8c22-110b-4dd3-a3b0-3687494fd1dc","Type":"ContainerStarted","Data":"fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0"} Nov 04 01:29:06.821886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.821768 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" event={"ID":"236a8c22-110b-4dd3-a3b0-3687494fd1dc","Type":"ContainerStarted","Data":"7d1289cc3fa5f607385cd0c86ce21bc01609906b2c3401f53bf87f5d4a2f7bcd"} Nov 04 01:29:06.821886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.821809 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:06.840400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.840359 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" podStartSLOduration=1.840323776 podStartE2EDuration="1.840323776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:06.838591416 +0000 UTC m=+1578.861523440" watchObservedRunningTime="2026-11-04 01:29:06.840323776 +0000 UTC m=+1578.863255783" Nov 04 01:29:06.861043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:06.861003 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" podStartSLOduration=1.8609908640000001 podStartE2EDuration="1.860990864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:06.859267814 +0000 UTC m=+1578.882199822" watchObservedRunningTime="2026-11-04 01:29:06.860990864 +0000 UTC m=+1578.883922870" Nov 04 01:29:07.001304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:07.001272 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27"] Nov 04 01:29:07.012301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:07.012271 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn"] Nov 04 01:29:08.113918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:08.113881 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc63742-f448-4195-bbe2-d15647027916" path="/var/lib/kubelet/pods/abc63742-f448-4195-bbe2-d15647027916/volumes" Nov 04 01:29:08.350587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:08.350556 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d"] Nov 04 01:29:08.369129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:08.369050 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph"] Nov 04 01:29:10.995467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:10.995433 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z"] Nov 04 01:29:10.996666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:10.995982 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" podUID="2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" containerName="maas-api" containerID="cri-o://11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41" gracePeriod=30 Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.270550 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x"] Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.271906 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.271930 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.271979 4312 cpu_manager.go:414] "containerMap: removing container" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.271989 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272006 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272015 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272035 4312 cpu_manager.go:414] "containerMap: removing container" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272061 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272081 4312 cpu_manager.go:414] "containerMap: removing container" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272088 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272300 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272315 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" Nov 04 01:29:11.276235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.272327 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:11.291709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.289795 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x"] Nov 04 01:29:11.291709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.289944 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.296185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.295694 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a362b6-serving-cert\"" Nov 04 01:29:11.353417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.353394 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:29:11.361949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.361918 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-metrics-certs\") pod \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " Nov 04 01:29:11.363601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.362009 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-maas-api-tls\") pod \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " Nov 04 01:29:11.363601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.362097 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-kube-api-access-9cqsb\" (UniqueName: \"kubernetes.io/projected/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-kube-api-access-9cqsb\") pod \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " Nov 04 01:29:11.363601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.362143 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-openshift-service-ca\") pod \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\" (UID: \"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41\") " Nov 04 01:29:11.363601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.362574 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7xv\" (UniqueName: \"kubernetes.io/projected/9a85f039-b6f3-4686-9529-115a20f63093-kube-api-access-xw7xv\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.363601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.362761 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a85f039-b6f3-4686-9529-115a20f63093-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.363601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.362836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.363601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.362917 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-metrics-certs\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.365271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.365243 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-openshift-service-ca" pod "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" (UID: "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:11.366049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.366016 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-metrics-certs" pod "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" (UID: "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:11.370149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.370118 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-maas-api-tls" pod "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" (UID: "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:11.370424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.370381 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-kube-api-access-9cqsb" pod "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" (UID: "2a6e6b00-37a7-41f6-8b73-0da04ecb6b41"). InnerVolumeSpecName "kube-api-access-9cqsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:11.464191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464152 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7xv\" (UniqueName: \"kubernetes.io/projected/9a85f039-b6f3-4686-9529-115a20f63093-kube-api-access-xw7xv\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.464428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464259 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a85f039-b6f3-4686-9529-115a20f63093-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.464428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464290 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.464569 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:11.464430 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-serving-cert: secret "maas-api-e2e-inf-a-a362b6-serving-cert" not found Nov 04 01:29:11.464569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464440 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-metrics-certs\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.464569 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:11.464480 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls podName:9a85f039-b6f3-4686-9529-115a20f63093 nodeName:}" failed. No retries permitted until 2026-11-04 01:29:11.964465081 +0000 UTC m=+1583.987397066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls") pod "maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" (UID: "9a85f039-b6f3-4686-9529-115a20f63093") : secret "maas-api-e2e-inf-a-a362b6-serving-cert" not found Nov 04 01:29:11.464695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464655 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:11.464695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464674 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:11.464695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464688 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cqsb\" (UniqueName: \"kubernetes.io/projected/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-kube-api-access-9cqsb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:11.464795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464697 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:11.464927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.464909 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a85f039-b6f3-4686-9529-115a20f63093-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.466805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.466788 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-metrics-certs\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.476795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.476770 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7xv\" (UniqueName: \"kubernetes.io/projected/9a85f039-b6f3-4686-9529-115a20f63093-kube-api-access-xw7xv\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.880137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.880094 4312 generic.go:356] "Generic (PLEG): container finished" podID="2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" containerID="11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41" exitCode=0 Nov 04 01:29:11.880436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.880382 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" event={"ID":"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41","Type":"ContainerDied","Data":"11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41"} Nov 04 01:29:11.880833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.880695 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" Nov 04 01:29:11.880966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.880820 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z" event={"ID":"2a6e6b00-37a7-41f6-8b73-0da04ecb6b41","Type":"ContainerDied","Data":"86844ca082410ddc75536440e699fd6ef2acb21eac6b4fe0fcefc85b4d85f99b"} Nov 04 01:29:11.881094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.881079 4312 scope.go:122] "RemoveContainer" containerID="11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41" Nov 04 01:29:11.897150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.897127 4312 scope.go:122] "RemoveContainer" containerID="11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41" Nov 04 01:29:11.897509 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:11.897488 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41\": container with ID starting with 11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41 not found: ID does not exist" containerID="11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41" Nov 04 01:29:11.897557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.897518 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41"} err="failed to get container status \"11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41\": rpc error: code = NotFound desc = could not find container \"11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41\": container with ID starting with 11dde95af9731718e2aab1a420271ce4566583c4df16f306fbf2e0e01c394f41 not found: ID does not exist" Nov 04 01:29:11.914626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.914585 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z"] Nov 04 01:29:11.920179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.920156 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e0642-7f66948678-98t5z"] Nov 04 01:29:11.974111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.974017 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:11.977246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:11.977215 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls\") pod \"maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:12.002083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.002032 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" podUID="6c5e7b78-7288-466e-b786-4a54459e6fef" containerName="payload-processing" containerID="cri-o://1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209" gracePeriod=25 Nov 04 01:29:12.013578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.013548 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" podUID="aa9f1ed9-7e45-4918-ac73-075ef310e33d" containerName="payload-pre-processing" containerID="cri-o://e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704" gracePeriod=25 Nov 04 01:29:12.111205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.111173 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" path="/var/lib/kubelet/pods/2a6e6b00-37a7-41f6-8b73-0da04ecb6b41/volumes" Nov 04 01:29:12.223488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.217945 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:12.367263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.367237 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:29:12.384966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.384708 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:29:12.414317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.414283 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9f1ed9-7e45-4918-ac73-075ef310e33d-kube-api-access-mb64r\" (UniqueName: \"kubernetes.io/projected/aa9f1ed9-7e45-4918-ac73-075ef310e33d-kube-api-access-mb64r\") pod \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " Nov 04 01:29:12.420683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.420322 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5e7b78-7288-466e-b786-4a54459e6fef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5e7b78-7288-466e-b786-4a54459e6fef-plugins-config-volume\") pod \"6c5e7b78-7288-466e-b786-4a54459e6fef\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " Nov 04 01:29:12.421230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.421203 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e7b78-7288-466e-b786-4a54459e6fef-plugins-config-volume" pod "6c5e7b78-7288-466e-b786-4a54459e6fef" (UID: "6c5e7b78-7288-466e-b786-4a54459e6fef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:12.421373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.421224 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9f1ed9-7e45-4918-ac73-075ef310e33d-plugins-config-volume" pod "aa9f1ed9-7e45-4918-ac73-075ef310e33d" (UID: "aa9f1ed9-7e45-4918-ac73-075ef310e33d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:12.421373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.421248 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9f1ed9-7e45-4918-ac73-075ef310e33d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1ed9-7e45-4918-ac73-075ef310e33d-plugins-config-volume\") pod \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\" (UID: \"aa9f1ed9-7e45-4918-ac73-075ef310e33d\") " Nov 04 01:29:12.421373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.421315 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5e7b78-7288-466e-b786-4a54459e6fef-kube-api-access-4x8gc\" (UniqueName: \"kubernetes.io/projected/6c5e7b78-7288-466e-b786-4a54459e6fef-kube-api-access-4x8gc\") pod \"6c5e7b78-7288-466e-b786-4a54459e6fef\" (UID: \"6c5e7b78-7288-466e-b786-4a54459e6fef\") " Nov 04 01:29:12.421910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.421891 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5e7b78-7288-466e-b786-4a54459e6fef-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:12.421992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.421915 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1ed9-7e45-4918-ac73-075ef310e33d-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:12.425796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.425770 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5e7b78-7288-466e-b786-4a54459e6fef-kube-api-access-4x8gc" pod "6c5e7b78-7288-466e-b786-4a54459e6fef" (UID: "6c5e7b78-7288-466e-b786-4a54459e6fef"). InnerVolumeSpecName "kube-api-access-4x8gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:12.429036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.429007 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9f1ed9-7e45-4918-ac73-075ef310e33d-kube-api-access-mb64r" pod "aa9f1ed9-7e45-4918-ac73-075ef310e33d" (UID: "aa9f1ed9-7e45-4918-ac73-075ef310e33d"). InnerVolumeSpecName "kube-api-access-mb64r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:12.522996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.522955 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb64r\" (UniqueName: \"kubernetes.io/projected/aa9f1ed9-7e45-4918-ac73-075ef310e33d-kube-api-access-mb64r\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:12.522996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.522988 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8gc\" (UniqueName: \"kubernetes.io/projected/6c5e7b78-7288-466e-b786-4a54459e6fef-kube-api-access-4x8gc\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:12.601568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.601433 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x"] Nov 04 01:29:12.889560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.889530 4312 generic.go:356] "Generic (PLEG): container finished" podID="6c5e7b78-7288-466e-b786-4a54459e6fef" containerID="1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209" exitCode=0 Nov 04 01:29:12.889722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.889565 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" event={"ID":"6c5e7b78-7288-466e-b786-4a54459e6fef","Type":"ContainerDied","Data":"1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209"} Nov 04 01:29:12.889722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.889606 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" Nov 04 01:29:12.889722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.889624 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" event={"ID":"6c5e7b78-7288-466e-b786-4a54459e6fef","Type":"ContainerDied","Data":"e629ca7461e26aaea9c4899096874ea779bc71a9723627b554364911016d012e"} Nov 04 01:29:12.889722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.889653 4312 scope.go:122] "RemoveContainer" containerID="1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209" Nov 04 01:29:12.892249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.892223 4312 generic.go:356] "Generic (PLEG): container finished" podID="aa9f1ed9-7e45-4918-ac73-075ef310e33d" containerID="e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704" exitCode=0 Nov 04 01:29:12.892396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.892261 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" Nov 04 01:29:12.892396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.892282 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" event={"ID":"aa9f1ed9-7e45-4918-ac73-075ef310e33d","Type":"ContainerDied","Data":"e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704"} Nov 04 01:29:12.892396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.892329 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" event={"ID":"aa9f1ed9-7e45-4918-ac73-075ef310e33d","Type":"ContainerDied","Data":"2ae976862a685b9eaaaeeb8e12cefc3ae82fcda87c99b0158f35174d86639e33"} Nov 04 01:29:12.893930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.893896 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" event={"ID":"9a85f039-b6f3-4686-9529-115a20f63093","Type":"ContainerStarted","Data":"926d8a64162a8853ea7318d9229b1c6fc2cb8d3f32b4cbfc90de9740b5a344f8"} Nov 04 01:29:12.901183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.901136 4312 scope.go:122] "RemoveContainer" containerID="1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209" Nov 04 01:29:12.901476 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:12.901454 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209\": container with ID starting with 1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209 not found: ID does not exist" containerID="1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209" Nov 04 01:29:12.901563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.901486 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209"} err="failed to get container status \"1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209\": rpc error: code = NotFound desc = could not find container \"1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209\": container with ID starting with 1b7255b1f3c275c87e040b8e9f8a98ff2472a271f4c16f065b90c128efaa4209 not found: ID does not exist" Nov 04 01:29:12.901563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.901509 4312 scope.go:122] "RemoveContainer" containerID="e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704" Nov 04 01:29:12.910729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.910713 4312 scope.go:122] "RemoveContainer" containerID="e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704" Nov 04 01:29:12.910973 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:12.910952 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704\": container with ID starting with e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704 not found: ID does not exist" containerID="e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704" Nov 04 01:29:12.911027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.910983 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704"} err="failed to get container status \"e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704\": rpc error: code = NotFound desc = could not find container \"e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704\": container with ID starting with e1ae79f5ae6e5b2018ef6d8e80c5104e6b0fb33cd148344616e86ea45828a704 not found: ID does not exist" Nov 04 01:29:12.916720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.916699 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27"] Nov 04 01:29:12.922580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.922562 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27"] Nov 04 01:29:12.934761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.934740 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn"] Nov 04 01:29:12.937642 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:12.937622 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn"] Nov 04 01:29:13.251466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.251425 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ecd1f-698bcd8759-kvdzn" podUID="aa9f1ed9-7e45-4918-ac73-075ef310e33d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.155:50052\" within 1s: context deadline exceeded" Nov 04 01:29:13.251831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.251427 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-6ecd1f-755f855db-j8q27" podUID="6c5e7b78-7288-466e-b786-4a54459e6fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.154:50052\" within 1s: context deadline exceeded" Nov 04 01:29:13.251831 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:13.251502 4312 logging.go:55] [core] [Channel #3133 SubChannel #3134]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.155:50052", ServerName: "10.133.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.155:50052: operation was canceled" Nov 04 01:29:13.251831 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:13.251520 4312 logging.go:55] [core] [Channel #3131 SubChannel #3132]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.154:50052", ServerName: "10.133.0.154:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.154:50052: operation was canceled" Nov 04 01:29:13.351635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.351589 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" podUID="98695cdd-5134-493d-9a8b-c3ac7a826e13" containerName="payload-processing" containerID="cri-o://84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff" gracePeriod=25 Nov 04 01:29:13.370168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.370120 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" podUID="bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" containerName="payload-pre-processing" containerID="cri-o://647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8" gracePeriod=25 Nov 04 01:29:13.645520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.645494 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:29:13.664233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.664209 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:29:13.739516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.739430 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-plugins-config-volume\") pod \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " Nov 04 01:29:13.739516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.739471 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98695cdd-5134-493d-9a8b-c3ac7a826e13-kube-api-access-s8wfb\" (UniqueName: \"kubernetes.io/projected/98695cdd-5134-493d-9a8b-c3ac7a826e13-kube-api-access-s8wfb\") pod \"98695cdd-5134-493d-9a8b-c3ac7a826e13\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " Nov 04 01:29:13.739727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.739652 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-kube-api-access-ncmqr\" (UniqueName: \"kubernetes.io/projected/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-kube-api-access-ncmqr\") pod \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\" (UID: \"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5\") " Nov 04 01:29:13.739829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.739807 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98695cdd-5134-493d-9a8b-c3ac7a826e13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98695cdd-5134-493d-9a8b-c3ac7a826e13-plugins-config-volume\") pod \"98695cdd-5134-493d-9a8b-c3ac7a826e13\" (UID: \"98695cdd-5134-493d-9a8b-c3ac7a826e13\") " Nov 04 01:29:13.739908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.739877 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-plugins-config-volume" pod "bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" (UID: "bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:13.740268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.740223 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98695cdd-5134-493d-9a8b-c3ac7a826e13-plugins-config-volume" pod "98695cdd-5134-493d-9a8b-c3ac7a826e13" (UID: "98695cdd-5134-493d-9a8b-c3ac7a826e13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:13.740386 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.740360 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98695cdd-5134-493d-9a8b-c3ac7a826e13-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:13.740440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.740384 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:13.741907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.741882 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-kube-api-access-ncmqr" pod "bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" (UID: "bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5"). InnerVolumeSpecName "kube-api-access-ncmqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:13.742207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.742177 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98695cdd-5134-493d-9a8b-c3ac7a826e13-kube-api-access-s8wfb" pod "98695cdd-5134-493d-9a8b-c3ac7a826e13" (UID: "98695cdd-5134-493d-9a8b-c3ac7a826e13"). InnerVolumeSpecName "kube-api-access-s8wfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:13.841290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.841256 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8wfb\" (UniqueName: \"kubernetes.io/projected/98695cdd-5134-493d-9a8b-c3ac7a826e13-kube-api-access-s8wfb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:13.841290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.841286 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncmqr\" (UniqueName: \"kubernetes.io/projected/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5-kube-api-access-ncmqr\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:13.901785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.901753 4312 generic.go:356] "Generic (PLEG): container finished" podID="98695cdd-5134-493d-9a8b-c3ac7a826e13" containerID="84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff" exitCode=0 Nov 04 01:29:13.901955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.901800 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" Nov 04 01:29:13.901955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.901843 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" event={"ID":"98695cdd-5134-493d-9a8b-c3ac7a826e13","Type":"ContainerDied","Data":"84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff"} Nov 04 01:29:13.901955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.901906 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d" event={"ID":"98695cdd-5134-493d-9a8b-c3ac7a826e13","Type":"ContainerDied","Data":"eb462968a89ee61a9e962282567616299c675830038d5fbad10bb5a2a51bef27"} Nov 04 01:29:13.901955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.901934 4312 scope.go:122] "RemoveContainer" containerID="84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff" Nov 04 01:29:13.903301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.903281 4312 generic.go:356] "Generic (PLEG): container finished" podID="bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" containerID="647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8" exitCode=0 Nov 04 01:29:13.903423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.903323 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" Nov 04 01:29:13.903423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.903373 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" event={"ID":"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5","Type":"ContainerDied","Data":"647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8"} Nov 04 01:29:13.903540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.903425 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph" event={"ID":"bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5","Type":"ContainerDied","Data":"fbf5493de6c1976f36fd85cc386379eea83d8434b79491ff42707a601087bbdd"} Nov 04 01:29:13.907589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.907564 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" event={"ID":"9a85f039-b6f3-4686-9529-115a20f63093","Type":"ContainerStarted","Data":"f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa"} Nov 04 01:29:13.907694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.907676 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:13.913026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.913005 4312 scope.go:122] "RemoveContainer" containerID="84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff" Nov 04 01:29:13.913328 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:13.913306 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff\": container with ID starting with 84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff not found: ID does not exist" containerID="84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff" Nov 04 01:29:13.913419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.913368 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff"} err="failed to get container status \"84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff\": rpc error: code = NotFound desc = could not find container \"84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff\": container with ID starting with 84dee2ea334a8eecf3f8e1776a1f028d69050bd1b0b3cc01fc3346bc9d869fff not found: ID does not exist" Nov 04 01:29:13.913419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.913392 4312 scope.go:122] "RemoveContainer" containerID="647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8" Nov 04 01:29:13.922166 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.922147 4312 scope.go:122] "RemoveContainer" containerID="647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8" Nov 04 01:29:13.922428 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:13.922409 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8\": container with ID starting with 647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8 not found: ID does not exist" containerID="647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8" Nov 04 01:29:13.922477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.922436 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8"} err="failed to get container status \"647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8\": rpc error: code = NotFound desc = could not find container \"647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8\": container with ID starting with 647a5525d9e932b30dabf35083f0a2f5b4256e2e4c652822d419a49b23690ff8 not found: ID does not exist" Nov 04 01:29:13.929987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.929944 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" podStartSLOduration=2.449033137 podStartE2EDuration="2.929930776s" totalImagesPullingTime="480.897639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:13.929300205 +0000 UTC m=+1585.952232231" watchObservedRunningTime="2026-11-04 01:29:13.929930776 +0000 UTC m=+1585.952862787" Nov 04 01:29:13.945110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.945089 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d"] Nov 04 01:29:13.949060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.949040 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e0642-7f599f57bd-kcr5d"] Nov 04 01:29:13.959842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.959819 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph"] Nov 04 01:29:13.964218 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:13.964198 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e0642-548f86dd9f-cmhph"] Nov 04 01:29:14.109889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:14.109848 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5e7b78-7288-466e-b786-4a54459e6fef" path="/var/lib/kubelet/pods/6c5e7b78-7288-466e-b786-4a54459e6fef/volumes" Nov 04 01:29:14.110498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:14.110482 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98695cdd-5134-493d-9a8b-c3ac7a826e13" path="/var/lib/kubelet/pods/98695cdd-5134-493d-9a8b-c3ac7a826e13/volumes" Nov 04 01:29:14.110984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:14.110970 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9f1ed9-7e45-4918-ac73-075ef310e33d" path="/var/lib/kubelet/pods/aa9f1ed9-7e45-4918-ac73-075ef310e33d/volumes" Nov 04 01:29:14.111508 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:14.111493 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" path="/var/lib/kubelet/pods/bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5/volumes" Nov 04 01:29:16.675965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:16.675931 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx"] Nov 04 01:29:16.676384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:16.676199 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" containerID="cri-o://a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0" gracePeriod=30 Nov 04 01:29:17.835962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.835930 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:29:17.836468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.836132 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:29:17.980591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.980555 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw"] Nov 04 01:29:17.981464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981442 4312 cpu_manager.go:414] "containerMap: removing container" podUID="98695cdd-5134-493d-9a8b-c3ac7a826e13" containerName="payload-processing" Nov 04 01:29:17.981464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981463 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98695cdd-5134-493d-9a8b-c3ac7a826e13" containerName="payload-processing" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981479 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" containerName="maas-api" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981488 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" containerName="maas-api" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981501 4312 cpu_manager.go:414] "containerMap: removing container" podUID="aa9f1ed9-7e45-4918-ac73-075ef310e33d" containerName="payload-pre-processing" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981509 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9f1ed9-7e45-4918-ac73-075ef310e33d" containerName="payload-pre-processing" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981542 4312 cpu_manager.go:414] "containerMap: removing container" podUID="bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" containerName="payload-pre-processing" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981550 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" containerName="payload-pre-processing" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981575 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6c5e7b78-7288-466e-b786-4a54459e6fef" containerName="payload-processing" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981583 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5e7b78-7288-466e-b786-4a54459e6fef" containerName="payload-processing" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981721 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdd5980-cd0c-4f7c-9e15-4b2b6d828910" containerName="istio-proxy" Nov 04 01:29:17.981735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981737 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc63742-f448-4195-bbe2-d15647027916" containerName="istio-proxy" Nov 04 01:29:17.982241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981746 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="98695cdd-5134-493d-9a8b-c3ac7a826e13" containerName="payload-processing" Nov 04 01:29:17.982241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981755 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6e6b00-37a7-41f6-8b73-0da04ecb6b41" containerName="maas-api" Nov 04 01:29:17.982241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981764 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9f1ed9-7e45-4918-ac73-075ef310e33d" containerName="payload-pre-processing" Nov 04 01:29:17.982241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981777 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf08c96c-4e00-40b6-8bf0-9d2795eb5fc5" containerName="payload-pre-processing" Nov 04 01:29:17.982241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.981788 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5e7b78-7288-466e-b786-4a54459e6fef" containerName="payload-processing" Nov 04 01:29:17.987397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.987371 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:17.997693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.996134 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8974fa-openshift-default-dockercfg-h4tg2\"" Nov 04 01:29:17.999523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:17.999326 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw"] Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.132573 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d807e2eb-432b-4219-814d-d5bbd21a4c53-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.132722 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.132767 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.132866 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.132902 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.132962 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.132990 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.133035 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrbm\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-kube-api-access-svrbm\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.144912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.133086 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.234780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.234744 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.234972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.234818 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d807e2eb-432b-4219-814d-d5bbd21a4c53-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.234972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.234941 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.234991 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235098 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235134 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235197 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235227 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235264 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235274 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrbm\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-kube-api-access-svrbm\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235655 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d807e2eb-432b-4219-814d-d5bbd21a4c53-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.235962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.235936 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.236276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.236253 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.236462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.236316 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.239328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.239301 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.240058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.240033 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.247729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.247704 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.247919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.247900 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrbm\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-kube-api-access-svrbm\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.311254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.311220 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:18.606109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.606080 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw"] Nov 04 01:29:18.606700 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:18.606670 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd807e2eb_432b_4219_814d_d5bbd21a4c53.slice/crio-499b0dd0f483285c12ff6f426b1bcc4927b4a55c0dfa3fe47b050848634972d8 WatchSource:0}: Error finding container 499b0dd0f483285c12ff6f426b1bcc4927b4a55c0dfa3fe47b050848634972d8: Status 404 returned error can't find the container with id 499b0dd0f483285c12ff6f426b1bcc4927b4a55c0dfa3fe47b050848634972d8 Nov 04 01:29:18.609372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.609309 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:18.609492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.609426 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:18.609492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.609474 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:18.950064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.949958 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" event={"ID":"d807e2eb-432b-4219-814d-d5bbd21a4c53","Type":"ContainerStarted","Data":"6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b"} Nov 04 01:29:18.950064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.950025 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" event={"ID":"d807e2eb-432b-4219-814d-d5bbd21a4c53","Type":"ContainerStarted","Data":"499b0dd0f483285c12ff6f426b1bcc4927b4a55c0dfa3fe47b050848634972d8"} Nov 04 01:29:18.977638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:18.977576 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podStartSLOduration=1.97755927 podStartE2EDuration="1.97755927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:18.974380484 +0000 UTC m=+1590.997312513" watchObservedRunningTime="2026-11-04 01:29:18.97755927 +0000 UTC m=+1591.000491277" Nov 04 01:29:19.320848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:19.320817 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:19.357855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:19.357820 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:19.634701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:19.634603 4312 patch_prober.go:29] interesting pod/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:29:19.634972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:19.634944 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:29:19.927698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:19.927614 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:29:19.981503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:19.981020 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:19.982645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:19.982615 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:21.982036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:21.982010 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:29:21.995635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:21.995607 4312 generic.go:356] "Generic (PLEG): container finished" podID="873797b0-f49c-467f-8157-1fccaf30b023" containerID="a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0" exitCode=0 Nov 04 01:29:21.995786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:21.995660 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" Nov 04 01:29:21.995786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:21.995692 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" event={"ID":"873797b0-f49c-467f-8157-1fccaf30b023","Type":"ContainerDied","Data":"a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0"} Nov 04 01:29:21.995786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:21.995745 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx" event={"ID":"873797b0-f49c-467f-8157-1fccaf30b023","Type":"ContainerDied","Data":"990a929ae5e86738d82cf3669fbe46970a2569fdf79c969ad4e641300d9cb3c1"} Nov 04 01:29:21.995786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:21.995767 4312 scope.go:122] "RemoveContainer" containerID="a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0" Nov 04 01:29:22.015399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.006614 4312 scope.go:122] "RemoveContainer" containerID="d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea" Nov 04 01:29:22.118891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.118833 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-kube-api-access-rx6v2\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-kube-api-access-rx6v2\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.143969 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/873797b0-f49c-467f-8157-1fccaf30b023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873797b0-f49c-467f-8157-1fccaf30b023-istio-podinfo\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.144023 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873797b0-f49c-467f-8157-1fccaf30b023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873797b0-f49c-467f-8157-1fccaf30b023-istiod-ca-cert\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.144052 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-data\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.144076 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-certs\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.144133 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-socket\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.144236 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-istio-token\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-istio-token\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.144254 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-credential-socket\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.144343 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-envoy\") pod \"873797b0-f49c-467f-8157-1fccaf30b023\" (UID: \"873797b0-f49c-467f-8157-1fccaf30b023\") " Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.145777 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-data" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.146086 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873797b0-f49c-467f-8157-1fccaf30b023-istiod-ca-cert" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:22.153361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.146580 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-credential-socket" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.175566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.175242 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/873797b0-f49c-467f-8157-1fccaf30b023-istio-podinfo" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:29:22.181358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.175930 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-socket" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.181358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.180439 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-certs" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.181765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.181737 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-kube-api-access-rx6v2" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "kube-api-access-rx6v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:22.181924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.181904 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-istio-token" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:22.185987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.184919 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-envoy" pod "873797b0-f49c-467f-8157-1fccaf30b023" (UID: "873797b0-f49c-467f-8157-1fccaf30b023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.191616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.191591 4312 scope.go:122] "RemoveContainer" containerID="a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0" Nov 04 01:29:22.191963 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:22.191932 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0\": container with ID starting with a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0 not found: ID does not exist" containerID="a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0" Nov 04 01:29:22.192006 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.191980 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0"} err="failed to get container status \"a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0\": rpc error: code = NotFound desc = could not find container \"a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0\": container with ID starting with a6aa9ff1acb81e09493c39932db1f33f1f0fffccc6265076e789280e38bfb6d0 not found: ID does not exist" Nov 04 01:29:22.192045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.192010 4312 scope.go:122] "RemoveContainer" containerID="d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea" Nov 04 01:29:22.192273 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:22.192257 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea\": container with ID starting with d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea not found: ID does not exist" containerID="d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea" Nov 04 01:29:22.192311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.192283 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea"} err="failed to get container status \"d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea\": rpc error: code = NotFound desc = could not find container \"d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea\": container with ID starting with d84f0c5ba8e300a784432bce5a04f2b0cb63cad4f70d03b0d3c1b056936112ea not found: ID does not exist" Nov 04 01:29:22.246084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.245999 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246034 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246050 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246067 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx6v2\" (UniqueName: \"kubernetes.io/projected/873797b0-f49c-467f-8157-1fccaf30b023-kube-api-access-rx6v2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246084 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873797b0-f49c-467f-8157-1fccaf30b023-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246099 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873797b0-f49c-467f-8157-1fccaf30b023-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246183 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246206 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.246381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.246219 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873797b0-f49c-467f-8157-1fccaf30b023-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.314517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.314475 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx"] Nov 04 01:29:22.319804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.319768 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1e0642-openshift-default-778b777787-hjhrx"] Nov 04 01:29:22.697555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.697518 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v"] Nov 04 01:29:22.698219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.698198 4312 cpu_manager.go:414] "containerMap: removing container" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" Nov 04 01:29:22.698368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.698219 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" Nov 04 01:29:22.698368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.698239 4312 cpu_manager.go:414] "containerMap: removing container" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" Nov 04 01:29:22.698368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.698247 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" Nov 04 01:29:22.698526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.698477 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" Nov 04 01:29:22.698526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.698492 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="873797b0-f49c-467f-8157-1fccaf30b023" containerName="istio-proxy" Nov 04 01:29:22.703471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.703449 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:22.711509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.711487 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8974fa-dockercfg-4zf4q\"" Nov 04 01:29:22.713237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.713217 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8974fa\"" Nov 04 01:29:22.719778 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.719753 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v"] Nov 04 01:29:22.739253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.739219 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb"] Nov 04 01:29:22.744592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.744570 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:22.782123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.782087 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb"] Nov 04 01:29:22.783194 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.783157 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxsp\" (UniqueName: \"kubernetes.io/projected/a7ed16d8-1905-44ae-8f12-df385a63dd88-kube-api-access-tcxsp\") pod \"payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:22.783516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.783482 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed16d8-1905-44ae-8f12-df385a63dd88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:22.904612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.900263 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt66t\" (UniqueName: \"kubernetes.io/projected/f5b07a71-14ba-435f-bf23-c55c6915c384-kube-api-access-rt66t\") pod \"payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:22.904612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.900541 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxsp\" (UniqueName: \"kubernetes.io/projected/a7ed16d8-1905-44ae-8f12-df385a63dd88-kube-api-access-tcxsp\") pod \"payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:22.904612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.901272 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed16d8-1905-44ae-8f12-df385a63dd88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:22.904612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.901325 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b07a71-14ba-435f-bf23-c55c6915c384-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:22.904612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.902609 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed16d8-1905-44ae-8f12-df385a63dd88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:22.931196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:22.930506 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxsp\" (UniqueName: \"kubernetes.io/projected/a7ed16d8-1905-44ae-8f12-df385a63dd88-kube-api-access-tcxsp\") pod \"payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:23.002456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.002361 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b07a71-14ba-435f-bf23-c55c6915c384-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:23.002954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.002474 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt66t\" (UniqueName: \"kubernetes.io/projected/f5b07a71-14ba-435f-bf23-c55c6915c384-kube-api-access-rt66t\") pod \"payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:23.003139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.003118 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b07a71-14ba-435f-bf23-c55c6915c384-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:23.015215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.015190 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt66t\" (UniqueName: \"kubernetes.io/projected/f5b07a71-14ba-435f-bf23-c55c6915c384-kube-api-access-rt66t\") pod \"payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:23.018095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.018073 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:23.126759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.125443 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:23.323562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.323533 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v"] Nov 04 01:29:23.324991 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:23.324966 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ed16d8_1905_44ae_8f12_df385a63dd88.slice/crio-b4000fc77f74475ba9f4cbe74a23f57fd645e4f0a2f20820bb8c9c84591854ba WatchSource:0}: Error finding container b4000fc77f74475ba9f4cbe74a23f57fd645e4f0a2f20820bb8c9c84591854ba: Status 404 returned error can't find the container with id b4000fc77f74475ba9f4cbe74a23f57fd645e4f0a2f20820bb8c9c84591854ba Nov 04 01:29:23.476552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:23.475687 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb"] Nov 04 01:29:24.012484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.012390 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" event={"ID":"f5b07a71-14ba-435f-bf23-c55c6915c384","Type":"ContainerStarted","Data":"f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe"} Nov 04 01:29:24.012484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.012473 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:24.012993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.012499 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" event={"ID":"f5b07a71-14ba-435f-bf23-c55c6915c384","Type":"ContainerStarted","Data":"dbded85416f31e27af47b9e55ef4ec807bf6cf77b8cebd0d13f2fd11865ba949"} Nov 04 01:29:24.014301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.014267 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" event={"ID":"a7ed16d8-1905-44ae-8f12-df385a63dd88","Type":"ContainerStarted","Data":"541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c"} Nov 04 01:29:24.014468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.014322 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" event={"ID":"a7ed16d8-1905-44ae-8f12-df385a63dd88","Type":"ContainerStarted","Data":"b4000fc77f74475ba9f4cbe74a23f57fd645e4f0a2f20820bb8c9c84591854ba"} Nov 04 01:29:24.014468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.014380 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:24.032961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.032908 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" podStartSLOduration=2.032894721 podStartE2EDuration="2.032894721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:24.030645309 +0000 UTC m=+1596.053577336" watchObservedRunningTime="2026-11-04 01:29:24.032894721 +0000 UTC m=+1596.055826758" Nov 04 01:29:24.064355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.052074 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" podStartSLOduration=2.052058094 podStartE2EDuration="2.052058094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:24.051683363 +0000 UTC m=+1596.074615394" watchObservedRunningTime="2026-11-04 01:29:24.052058094 +0000 UTC m=+1596.074990102" Nov 04 01:29:24.111112 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:24.111079 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873797b0-f49c-467f-8157-1fccaf30b023" path="/var/lib/kubelet/pods/873797b0-f49c-467f-8157-1fccaf30b023/volumes" Nov 04 01:29:27.444749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:27.444704 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n"] Nov 04 01:29:27.449169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:27.449138 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6"] Nov 04 01:29:28.129711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.129678 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf"] Nov 04 01:29:28.134937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.134912 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.142372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.140408 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8974fa-serving-cert\"" Nov 04 01:29:28.146628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.146602 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf"] Nov 04 01:29:28.292621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.292587 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757cedc8-09f9-4db9-9a99-72b42341e1f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.292804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.292638 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpn5\" (UniqueName: \"kubernetes.io/projected/757cedc8-09f9-4db9-9a99-72b42341e1f1-kube-api-access-sfpn5\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.292804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.292768 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-metrics-certs\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.292913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.292809 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.393959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.393880 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.394097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.394042 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757cedc8-09f9-4db9-9a99-72b42341e1f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.394097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.394078 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpn5\" (UniqueName: \"kubernetes.io/projected/757cedc8-09f9-4db9-9a99-72b42341e1f1-kube-api-access-sfpn5\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.394208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.394136 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-metrics-certs\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.394718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.394688 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757cedc8-09f9-4db9-9a99-72b42341e1f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.396582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.396561 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-metrics-certs\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.396715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.396638 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.406581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.406556 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpn5\" (UniqueName: \"kubernetes.io/projected/757cedc8-09f9-4db9-9a99-72b42341e1f1-kube-api-access-sfpn5\") pod \"maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.457469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.457437 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:28.745171 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:28.745138 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757cedc8_09f9_4db9_9a99_72b42341e1f1.slice/crio-abd8116d3420232713202b198ae208502a123622529f6e6b0750a3cc0c5f8087 WatchSource:0}: Error finding container abd8116d3420232713202b198ae208502a123622529f6e6b0750a3cc0c5f8087: Status 404 returned error can't find the container with id abd8116d3420232713202b198ae208502a123622529f6e6b0750a3cc0c5f8087 Nov 04 01:29:28.745469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:28.745437 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf"] Nov 04 01:29:29.096923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:29.096881 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" event={"ID":"757cedc8-09f9-4db9-9a99-72b42341e1f1","Type":"ContainerStarted","Data":"abd8116d3420232713202b198ae208502a123622529f6e6b0750a3cc0c5f8087"} Nov 04 01:29:29.112432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:29.112401 4312 generic.go:356] "Generic (PLEG): container finished" podID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerID="4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191" exitCode=0 Nov 04 01:29:29.113031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:29.113002 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" event={"ID":"4204c8e6-9476-4a48-a947-635f16cb21c7","Type":"ContainerDied","Data":"4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191"} Nov 04 01:29:29.113597 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:29.113576 4312 scope.go:122] "RemoveContainer" containerID="4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191" Nov 04 01:29:29.115876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:29.115838 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:29.116036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:29.115955 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:29.116036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:29.116000 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:29.459220 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:29.459176 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:29:30.121691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.121657 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" event={"ID":"757cedc8-09f9-4db9-9a99-72b42341e1f1","Type":"ContainerStarted","Data":"e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec"} Nov 04 01:29:30.121904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.121781 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:30.124760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.124732 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" event={"ID":"4204c8e6-9476-4a48-a947-635f16cb21c7","Type":"ContainerStarted","Data":"482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb"} Nov 04 01:29:30.145960 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.145911 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" podStartSLOduration=1.659940746 podStartE2EDuration="2.145893711s" totalImagesPullingTime="485.952965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:30.1442719 +0000 UTC m=+1602.167203926" watchObservedRunningTime="2026-11-04 01:29:30.145893711 +0000 UTC m=+1602.168825719" Nov 04 01:29:30.386413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.385651 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj"] Nov 04 01:29:30.391695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.391660 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.406281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.406253 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj"] Nov 04 01:29:30.414668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.412603 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-362631-openshift-default-dockercfg-2s9lc\"" Nov 04 01:29:30.525272 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525231 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-token\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525272 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525280 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-data\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525375 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-envoy\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525483 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istiod-ca-cert\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525584 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-certs\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525617 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-podinfo\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525656 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcmc\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-kube-api-access-wqcmc\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525715 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-socket\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.525829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.525764 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-credential-socket\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627132 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-certs\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627200 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-podinfo\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627260 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcmc\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-kube-api-access-wqcmc\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627297 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-socket\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627369 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-credential-socket\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627414 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-token\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627457 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-data\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627488 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-envoy\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.627602 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istiod-ca-cert\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.628388 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istiod-ca-cert\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.633353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.628634 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-data\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.646428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.637271 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-credential-socket\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.646428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.637536 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-certs\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.646428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.637796 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-socket\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.665372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.655031 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-token\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.665372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.657688 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-podinfo\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.681790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.681746 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-envoy\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.707531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.689227 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcmc\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-kube-api-access-wqcmc\") pod \"e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.714289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.714256 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:30.782847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.782817 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:30.784283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.784256 4312 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:29:30.784349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:30.784312 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:29:31.024207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.024177 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj"] Nov 04 01:29:31.024789 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:31.024761 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b3756c_b4a5_49f2_a3fb_539cbb63c9c4.slice/crio-cb314f82729636a2fa05a2b1dc0f66b1476a42fa36d030d3a8f6048f53aedb27 WatchSource:0}: Error finding container cb314f82729636a2fa05a2b1dc0f66b1476a42fa36d030d3a8f6048f53aedb27: Status 404 returned error can't find the container with id cb314f82729636a2fa05a2b1dc0f66b1476a42fa36d030d3a8f6048f53aedb27 Nov 04 01:29:31.027487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.027455 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:31.027568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.027521 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:31.027568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.027548 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:31.132746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.132711 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" event={"ID":"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4","Type":"ContainerStarted","Data":"771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280"} Nov 04 01:29:31.133020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.132980 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" event={"ID":"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4","Type":"ContainerStarted","Data":"cb314f82729636a2fa05a2b1dc0f66b1476a42fa36d030d3a8f6048f53aedb27"} Nov 04 01:29:31.714991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.714955 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:31.783597 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.783560 4312 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:29:31.783770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:31.783643 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:29:32.721191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:32.721158 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:32.747013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:32.746962 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podStartSLOduration=2.746948553 podStartE2EDuration="2.746948553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:31.18061443 +0000 UTC m=+1603.203546451" watchObservedRunningTime="2026-11-04 01:29:32.746948553 +0000 UTC m=+1604.769880556" Nov 04 01:29:32.782858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:32.782821 4312 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:29:32.783037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:32.782873 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:29:33.146317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:33.146290 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:33.158357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:33.147321 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:29:33.783386 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:33.783325 4312 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:29:33.783809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:33.783407 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:29:33.792724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:33.792697 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:34.782819 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:34.782786 4312 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:29:34.782988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:34.782837 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:29:35.029418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.029386 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:29:35.030032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.030014 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:29:35.273215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.273130 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt"] Nov 04 01:29:35.277700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.277659 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.285151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.285101 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-362631\"" Nov 04 01:29:35.285525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.285496 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-362631-dockercfg-ss42w\"" Nov 04 01:29:35.289711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.289683 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt"] Nov 04 01:29:35.293984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.293959 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb"] Nov 04 01:29:35.307846 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.307813 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.319760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.319727 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb"] Nov 04 01:29:35.375371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.375314 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb"] Nov 04 01:29:35.390308 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.390269 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2"] Nov 04 01:29:35.418962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.418919 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.420221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.419399 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwwp4\" (UniqueName: \"kubernetes.io/projected/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-kube-api-access-zwwp4\") pod \"payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.420477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.420450 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e130b444-8588-48ad-b422-54342eeae735-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.420704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.420679 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msv57\" (UniqueName: \"kubernetes.io/projected/e130b444-8588-48ad-b422-54342eeae735-kube-api-access-msv57\") pod \"payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.521745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.521712 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msv57\" (UniqueName: \"kubernetes.io/projected/e130b444-8588-48ad-b422-54342eeae735-kube-api-access-msv57\") pod \"payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.521933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.521834 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.521933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.521922 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwwp4\" (UniqueName: \"kubernetes.io/projected/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-kube-api-access-zwwp4\") pod \"payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.522025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.521976 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e130b444-8588-48ad-b422-54342eeae735-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.522658 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.522638 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.522710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.522666 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e130b444-8588-48ad-b422-54342eeae735-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.533944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.533918 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msv57\" (UniqueName: \"kubernetes.io/projected/e130b444-8588-48ad-b422-54342eeae735-kube-api-access-msv57\") pod \"payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.534100 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.534034 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwwp4\" (UniqueName: \"kubernetes.io/projected/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-kube-api-access-zwwp4\") pod \"payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.595440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.595408 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:35.637808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.633892 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:35.794033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.793957 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:35.796211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.796188 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:29:35.955803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:35.955772 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt"] Nov 04 01:29:35.956863 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:35.956839 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bd26f2_2287_4ae2_9640_dfbe6e1409ee.slice/crio-ca1e6edfec1ae484721d08b4fafe1a2cb34255ff30f7e47a2814623154cb6c60 WatchSource:0}: Error finding container ca1e6edfec1ae484721d08b4fafe1a2cb34255ff30f7e47a2814623154cb6c60: Status 404 returned error can't find the container with id ca1e6edfec1ae484721d08b4fafe1a2cb34255ff30f7e47a2814623154cb6c60 Nov 04 01:29:36.031510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.031485 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb"] Nov 04 01:29:36.031885 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:36.031498 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode130b444_8588_48ad_b422_54342eeae735.slice/crio-dfc16052aafa3268d461af19d0b7eff7cdbc24f9ec929e81536c6d0eb839c0e0 WatchSource:0}: Error finding container dfc16052aafa3268d461af19d0b7eff7cdbc24f9ec929e81536c6d0eb839c0e0: Status 404 returned error can't find the container with id dfc16052aafa3268d461af19d0b7eff7cdbc24f9ec929e81536c6d0eb839c0e0 Nov 04 01:29:36.138704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.138677 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:29:36.193430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.193386 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" event={"ID":"e130b444-8588-48ad-b422-54342eeae735","Type":"ContainerStarted","Data":"8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947"} Nov 04 01:29:36.193620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.193457 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" event={"ID":"e130b444-8588-48ad-b422-54342eeae735","Type":"ContainerStarted","Data":"dfc16052aafa3268d461af19d0b7eff7cdbc24f9ec929e81536c6d0eb839c0e0"} Nov 04 01:29:36.193807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.193790 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:36.195318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.195288 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" event={"ID":"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee","Type":"ContainerStarted","Data":"b4cbe68599caa72570d267460eea088bd8eef2b2523592a4017e7e974f6e3fe8"} Nov 04 01:29:36.195467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.195372 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" event={"ID":"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee","Type":"ContainerStarted","Data":"ca1e6edfec1ae484721d08b4fafe1a2cb34255ff30f7e47a2814623154cb6c60"} Nov 04 01:29:36.195669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.195648 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:36.223608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.223534 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" podStartSLOduration=1.223512707 podStartE2EDuration="1.223512707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:36.21755556 +0000 UTC m=+1608.240487568" watchObservedRunningTime="2026-11-04 01:29:36.223512707 +0000 UTC m=+1608.246444717" Nov 04 01:29:36.243512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:36.243460 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" podStartSLOduration=1.243445017 podStartE2EDuration="1.243445017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:36.238509872 +0000 UTC m=+1608.261441896" watchObservedRunningTime="2026-11-04 01:29:36.243445017 +0000 UTC m=+1608.266377032" Nov 04 01:29:40.375950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.375906 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" podUID="1c87cbe9-94fa-479e-bb0d-38ad669eb71f" containerName="payload-processing" containerID="cri-o://cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7" gracePeriod=25 Nov 04 01:29:40.391149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.391115 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" podUID="461d2a93-be30-4606-932d-bc5c95097a64" containerName="payload-pre-processing" containerID="cri-o://ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173" gracePeriod=25 Nov 04 01:29:40.669394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.669365 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:29:40.808672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.808635 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-kube-api-access-xwzts\" (UniqueName: \"kubernetes.io/projected/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-kube-api-access-xwzts\") pod \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " Nov 04 01:29:40.808870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.808729 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-plugins-config-volume\") pod \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\" (UID: \"1c87cbe9-94fa-479e-bb0d-38ad669eb71f\") " Nov 04 01:29:40.809066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.809043 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-plugins-config-volume" pod "1c87cbe9-94fa-479e-bb0d-38ad669eb71f" (UID: "1c87cbe9-94fa-479e-bb0d-38ad669eb71f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:40.809279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.809250 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:40.810834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.810813 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-kube-api-access-xwzts" pod "1c87cbe9-94fa-479e-bb0d-38ad669eb71f" (UID: "1c87cbe9-94fa-479e-bb0d-38ad669eb71f"). InnerVolumeSpecName "kube-api-access-xwzts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:40.914411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:40.911561 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzts\" (UniqueName: \"kubernetes.io/projected/1c87cbe9-94fa-479e-bb0d-38ad669eb71f-kube-api-access-xwzts\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.236779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.236690 4312 generic.go:356] "Generic (PLEG): container finished" podID="1c87cbe9-94fa-479e-bb0d-38ad669eb71f" containerID="cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7" exitCode=0 Nov 04 01:29:41.236955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.236873 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" event={"ID":"1c87cbe9-94fa-479e-bb0d-38ad669eb71f","Type":"ContainerDied","Data":"cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7"} Nov 04 01:29:41.236955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.236925 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" event={"ID":"1c87cbe9-94fa-479e-bb0d-38ad669eb71f","Type":"ContainerDied","Data":"e44efac67bc59c33ef14380bbc25c902dfcc4260f03016203e9206b0f96cb691"} Nov 04 01:29:41.236955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.236954 4312 scope.go:122] "RemoveContainer" containerID="cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7" Nov 04 01:29:41.237164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.237127 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb" Nov 04 01:29:41.252167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.252143 4312 scope.go:122] "RemoveContainer" containerID="cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7" Nov 04 01:29:41.252473 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:41.252453 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7\": container with ID starting with cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7 not found: ID does not exist" containerID="cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7" Nov 04 01:29:41.252548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.252486 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7"} err="failed to get container status \"cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7\": rpc error: code = NotFound desc = could not find container \"cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7\": container with ID starting with cb935c17b893ba62e9d77e1475555bdccbf8620a61f65ffc38d10740c54a25d7 not found: ID does not exist" Nov 04 01:29:41.268984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.268957 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb"] Nov 04 01:29:41.277665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.277639 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a5115a-7dc9bc97b-ww8fb"] Nov 04 01:29:41.416180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.416148 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf"] Nov 04 01:29:41.417170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.417139 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1c87cbe9-94fa-479e-bb0d-38ad669eb71f" containerName="payload-processing" Nov 04 01:29:41.417170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.417171 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c87cbe9-94fa-479e-bb0d-38ad669eb71f" containerName="payload-processing" Nov 04 01:29:41.417403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.417323 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c87cbe9-94fa-479e-bb0d-38ad669eb71f" containerName="payload-processing" Nov 04 01:29:41.422699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.422677 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.444700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.444670 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf"] Nov 04 01:29:41.462591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.462543 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-362631-serving-cert\"" Nov 04 01:29:41.520985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.520879 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95r6\" (UniqueName: \"kubernetes.io/projected/fd92106e-d547-4052-a6a0-de9222fa375a-kube-api-access-v95r6\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.521149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.520999 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-maas-api-tls\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.521149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.521134 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-metrics-certs\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.521297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.521190 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd92106e-d547-4052-a6a0-de9222fa375a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.623096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.623055 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-maas-api-tls\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.623292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.623196 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-metrics-certs\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.623292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.623262 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd92106e-d547-4052-a6a0-de9222fa375a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.623448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.623429 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95r6\" (UniqueName: \"kubernetes.io/projected/fd92106e-d547-4052-a6a0-de9222fa375a-kube-api-access-v95r6\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.624749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.624712 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd92106e-d547-4052-a6a0-de9222fa375a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.626486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.626450 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-maas-api-tls\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.627708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.627662 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-metrics-certs\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.636623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.636571 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95r6\" (UniqueName: \"kubernetes.io/projected/fd92106e-d547-4052-a6a0-de9222fa375a-kube-api-access-v95r6\") pod \"maas-api-e2e-inf-b-362631-8494697844-jsmqf\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:41.751305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:41.751263 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:42.399366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:42.398660 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c87cbe9-94fa-479e-bb0d-38ad669eb71f" path="/var/lib/kubelet/pods/1c87cbe9-94fa-479e-bb0d-38ad669eb71f/volumes" Nov 04 01:29:42.418032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:42.417989 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf"] Nov 04 01:29:42.427827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:42.426173 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" event={"ID":"fd92106e-d547-4052-a6a0-de9222fa375a","Type":"ContainerStarted","Data":"1a4e141ce9cf8d210b3c4df51a7628ee3c8a639b26622a892f2ed1a3897a6e85"} Nov 04 01:29:42.503414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:42.501730 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" containerID="cri-o://f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953" gracePeriod=45 Nov 04 01:29:42.503414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:42.501914 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" containerID="cri-o://d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea" gracePeriod=45 Nov 04 01:29:43.086368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.086309 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:29:43.113832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.113812 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:29:43.171354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.166007 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-model-cache\") pod \"a7aac864-b53f-411d-81ef-9e94e015cdad\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " Nov 04 01:29:43.171354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.166090 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7aac864-b53f-411d-81ef-9e94e015cdad-kube-api-access-rx5p2\" (UniqueName: \"kubernetes.io/projected/a7aac864-b53f-411d-81ef-9e94e015cdad-kube-api-access-rx5p2\") pod \"a7aac864-b53f-411d-81ef-9e94e015cdad\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " Nov 04 01:29:43.171354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.166187 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-dshm\") pod \"a7aac864-b53f-411d-81ef-9e94e015cdad\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " Nov 04 01:29:43.171354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.166237 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-tmp-dir\") pod \"a7aac864-b53f-411d-81ef-9e94e015cdad\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " Nov 04 01:29:43.171354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.166301 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7aac864-b53f-411d-81ef-9e94e015cdad-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aac864-b53f-411d-81ef-9e94e015cdad-tls-certs\") pod \"a7aac864-b53f-411d-81ef-9e94e015cdad\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " Nov 04 01:29:43.171354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.166417 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-home\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-home\") pod \"a7aac864-b53f-411d-81ef-9e94e015cdad\" (UID: \"a7aac864-b53f-411d-81ef-9e94e015cdad\") " Nov 04 01:29:43.171354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.167240 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-home" pod "a7aac864-b53f-411d-81ef-9e94e015cdad" (UID: "a7aac864-b53f-411d-81ef-9e94e015cdad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.194447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.194406 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-model-cache" pod "a7aac864-b53f-411d-81ef-9e94e015cdad" (UID: "a7aac864-b53f-411d-81ef-9e94e015cdad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.194447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.194429 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-tmp-dir" pod "a7aac864-b53f-411d-81ef-9e94e015cdad" (UID: "a7aac864-b53f-411d-81ef-9e94e015cdad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.199529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.199493 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-dshm" pod "a7aac864-b53f-411d-81ef-9e94e015cdad" (UID: "a7aac864-b53f-411d-81ef-9e94e015cdad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.199529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.199500 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7aac864-b53f-411d-81ef-9e94e015cdad-tls-certs" pod "a7aac864-b53f-411d-81ef-9e94e015cdad" (UID: "a7aac864-b53f-411d-81ef-9e94e015cdad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:43.201822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.201797 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7aac864-b53f-411d-81ef-9e94e015cdad-kube-api-access-rx5p2" pod "a7aac864-b53f-411d-81ef-9e94e015cdad" (UID: "a7aac864-b53f-411d-81ef-9e94e015cdad"). InnerVolumeSpecName "kube-api-access-rx5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.268803 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71320d68-38c3-4423-bf81-0949d491ba50-kube-api-access-ccsvd\" (UniqueName: \"kubernetes.io/projected/71320d68-38c3-4423-bf81-0949d491ba50-kube-api-access-ccsvd\") pod \"71320d68-38c3-4423-bf81-0949d491ba50\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.268866 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-home\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-home\") pod \"71320d68-38c3-4423-bf81-0949d491ba50\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.268918 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71320d68-38c3-4423-bf81-0949d491ba50-tls-certs\" (UniqueName: \"kubernetes.io/secret/71320d68-38c3-4423-bf81-0949d491ba50-tls-certs\") pod \"71320d68-38c3-4423-bf81-0949d491ba50\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.268986 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-tmp-dir\") pod \"71320d68-38c3-4423-bf81-0949d491ba50\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269060 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-model-cache\") pod \"71320d68-38c3-4423-bf81-0949d491ba50\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269138 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-dshm\") pod \"71320d68-38c3-4423-bf81-0949d491ba50\" (UID: \"71320d68-38c3-4423-bf81-0949d491ba50\") " Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269759 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269782 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269794 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aac864-b53f-411d-81ef-9e94e015cdad-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269810 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269823 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aac864-b53f-411d-81ef-9e94e015cdad-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.270535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.269839 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5p2\" (UniqueName: \"kubernetes.io/projected/a7aac864-b53f-411d-81ef-9e94e015cdad-kube-api-access-rx5p2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.273815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.273780 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71320d68-38c3-4423-bf81-0949d491ba50-tls-certs" pod "71320d68-38c3-4423-bf81-0949d491ba50" (UID: "71320d68-38c3-4423-bf81-0949d491ba50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:43.274029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.273997 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-tmp-dir" pod "71320d68-38c3-4423-bf81-0949d491ba50" (UID: "71320d68-38c3-4423-bf81-0949d491ba50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.274169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.274034 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-home" pod "71320d68-38c3-4423-bf81-0949d491ba50" (UID: "71320d68-38c3-4423-bf81-0949d491ba50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.274411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.274374 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-model-cache" pod "71320d68-38c3-4423-bf81-0949d491ba50" (UID: "71320d68-38c3-4423-bf81-0949d491ba50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.274504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.274434 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-dshm" pod "71320d68-38c3-4423-bf81-0949d491ba50" (UID: "71320d68-38c3-4423-bf81-0949d491ba50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:43.276003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.275984 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71320d68-38c3-4423-bf81-0949d491ba50-kube-api-access-ccsvd" pod "71320d68-38c3-4423-bf81-0949d491ba50" (UID: "71320d68-38c3-4423-bf81-0949d491ba50"). InnerVolumeSpecName "kube-api-access-ccsvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:43.373486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.373449 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.373714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.373695 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccsvd\" (UniqueName: \"kubernetes.io/projected/71320d68-38c3-4423-bf81-0949d491ba50-kube-api-access-ccsvd\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.373816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.373803 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.373901 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.373891 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71320d68-38c3-4423-bf81-0949d491ba50-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.373991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.373980 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.374082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.374070 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71320d68-38c3-4423-bf81-0949d491ba50-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:43.441249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.441217 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" event={"ID":"fd92106e-d547-4052-a6a0-de9222fa375a","Type":"ContainerStarted","Data":"79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c"} Nov 04 01:29:43.441667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.441280 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:43.442716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.442692 4312 generic.go:356] "Generic (PLEG): container finished" podID="71320d68-38c3-4423-bf81-0949d491ba50" containerID="d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea" exitCode=0 Nov 04 01:29:43.442811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.442744 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" Nov 04 01:29:43.442811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.442778 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" event={"ID":"71320d68-38c3-4423-bf81-0949d491ba50","Type":"ContainerDied","Data":"d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea"} Nov 04 01:29:43.442922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.442814 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" event={"ID":"71320d68-38c3-4423-bf81-0949d491ba50","Type":"ContainerDied","Data":"808f716db814cf2866052ba94812cedf6d0746dfa3a2f5eba40050a4535c95dc"} Nov 04 01:29:43.442922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.442833 4312 scope.go:122] "RemoveContainer" containerID="d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea" Nov 04 01:29:43.444309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.444286 4312 generic.go:356] "Generic (PLEG): container finished" podID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerID="f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953" exitCode=0 Nov 04 01:29:43.444433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.444323 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" Nov 04 01:29:43.444433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.444363 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" event={"ID":"a7aac864-b53f-411d-81ef-9e94e015cdad","Type":"ContainerDied","Data":"f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953"} Nov 04 01:29:43.444433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.444403 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n" event={"ID":"a7aac864-b53f-411d-81ef-9e94e015cdad","Type":"ContainerDied","Data":"e1f2bb28e0b196c056e0a6eb0dceba8b15fa43d52b2034dc2a5b0b3345847db8"} Nov 04 01:29:43.465370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.464878 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" podStartSLOduration=1.823968286 podStartE2EDuration="2.464861912s" totalImagesPullingTime="640.893626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:43.463689543 +0000 UTC m=+1615.486621575" watchObservedRunningTime="2026-11-04 01:29:43.464861912 +0000 UTC m=+1615.487793920" Nov 04 01:29:43.479978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.479957 4312 scope.go:122] "RemoveContainer" containerID="d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea" Nov 04 01:29:43.480406 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:43.480380 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea\": container with ID starting with d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea not found: ID does not exist" containerID="d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea" Nov 04 01:29:43.480480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.480419 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea"} err="failed to get container status \"d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea\": rpc error: code = NotFound desc = could not find container \"d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea\": container with ID starting with d2696670f2d914ce73a52ca626a7d09f5ac407aeb3ba83d2059121ac06e9a7ea not found: ID does not exist" Nov 04 01:29:43.480480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.480444 4312 scope.go:122] "RemoveContainer" containerID="f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953" Nov 04 01:29:43.496599 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.496568 4312 scope.go:122] "RemoveContainer" containerID="f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953" Nov 04 01:29:43.497795 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:43.497520 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953\": container with ID starting with f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953 not found: ID does not exist" containerID="f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953" Nov 04 01:29:43.497795 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.497581 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953"} err="failed to get container status \"f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953\": rpc error: code = NotFound desc = could not find container \"f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953\": container with ID starting with f8731eb239aba7062a326e9736160bdb701e3f7c74349c58e29624fd86f80953 not found: ID does not exist" Nov 04 01:29:43.498738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.498716 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n"] Nov 04 01:29:43.502136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.502110 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-2-w3-a5115a-kserve-587596f695-tt29n"] Nov 04 01:29:43.512656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.512635 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6"] Nov 04 01:29:43.519951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:43.519928 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6"] Nov 04 01:29:44.022904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:44.022810 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/distinct-w3-a5115a-kserve-66b5dc8ff6-tr2v6" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:44.109454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:44.109423 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71320d68-38c3-4423-bf81-0949d491ba50" path="/var/lib/kubelet/pods/71320d68-38c3-4423-bf81-0949d491ba50/volumes" Nov 04 01:29:44.109842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:44.109828 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" path="/var/lib/kubelet/pods/a7aac864-b53f-411d-81ef-9e94e015cdad/volumes" Nov 04 01:29:47.206905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.206879 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:29:47.207442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.206957 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:29:47.975457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.975385 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj"] Nov 04 01:29:47.976095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.976079 4312 cpu_manager.go:414] "containerMap: removing container" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" Nov 04 01:29:47.976141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.976096 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" Nov 04 01:29:47.976141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.976124 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" Nov 04 01:29:47.976141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.976132 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" Nov 04 01:29:47.976260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.976250 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="71320d68-38c3-4423-bf81-0949d491ba50" containerName="main" Nov 04 01:29:47.976297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.976271 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7aac864-b53f-411d-81ef-9e94e015cdad" containerName="main" Nov 04 01:29:47.981289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.981273 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:47.987500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.987465 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2aefc5\"" Nov 04 01:29:47.992066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:47.992040 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj"] Nov 04 01:29:48.016397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.016364 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7"] Nov 04 01:29:48.022890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.022861 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7"] Nov 04 01:29:48.023050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.022973 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.029705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.029681 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2aefc5-dockercfg-jh8s4\"" Nov 04 01:29:48.039201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.039173 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e65f77-0ff8-420f-94a6-b029fb7916c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.039370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.039225 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwfgr\" (UniqueName: \"kubernetes.io/projected/0c9e25e2-d171-4142-82a6-a877a50cbea6-kube-api-access-mwfgr\") pod \"payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:48.039370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.039359 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt8q\" (UniqueName: \"kubernetes.io/projected/81e65f77-0ff8-420f-94a6-b029fb7916c9-kube-api-access-dnt8q\") pod \"payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.039545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.039520 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9e25e2-d171-4142-82a6-a877a50cbea6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:48.072547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.072512 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs"] Nov 04 01:29:48.077667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.077643 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.156400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.156353 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.156833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.156806 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.157036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.156983 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt8q\" (UniqueName: \"kubernetes.io/projected/81e65f77-0ff8-420f-94a6-b029fb7916c9-kube-api-access-dnt8q\") pod \"payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.157251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.157224 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pbw\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-kube-api-access-g7pbw\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.157391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.157376 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.157541 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.157528 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.157651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.157637 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.157745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.157734 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9e25e2-d171-4142-82a6-a877a50cbea6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:48.157932 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.157915 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.158033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.158021 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1cbc30-9a61-4274-b2a4-9da61d518423-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.158202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.158185 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e65f77-0ff8-420f-94a6-b029fb7916c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.193464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.162612 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e65f77-0ff8-420f-94a6-b029fb7916c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.193741 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.169648 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9e25e2-d171-4142-82a6-a877a50cbea6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:48.193912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.190652 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs"] Nov 04 01:29:48.194073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.194054 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.194168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.194094 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwfgr\" (UniqueName: \"kubernetes.io/projected/0c9e25e2-d171-4142-82a6-a877a50cbea6-kube-api-access-mwfgr\") pod \"payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:48.195094 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.195069 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt8q\" (UniqueName: \"kubernetes.io/projected/81e65f77-0ff8-420f-94a6-b029fb7916c9-kube-api-access-dnt8q\") pod \"payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.246088 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.245977 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwfgr\" (UniqueName: \"kubernetes.io/projected/0c9e25e2-d171-4142-82a6-a877a50cbea6-kube-api-access-mwfgr\") pod \"payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:48.295429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295389 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.295617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295510 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.295617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295559 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.295755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295634 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pbw\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-kube-api-access-g7pbw\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.295755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295680 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.295867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295806 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.295922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295866 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.295975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.295919 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.296068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.296047 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.296147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.296128 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.296215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.296171 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1cbc30-9a61-4274-b2a4-9da61d518423-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.296451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.296429 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.296580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.296559 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.296942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.296910 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1cbc30-9a61-4274-b2a4-9da61d518423-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.297953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.297923 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.298075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.297926 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:48.298395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.298378 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.308441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.308419 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pbw\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-kube-api-access-g7pbw\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.308785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.308761 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.346878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.346850 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2aefc5-dockercfg-jh8s4\"" Nov 04 01:29:48.355511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.355461 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:48.510471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.510382 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:48.590889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.590857 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj"] Nov 04 01:29:48.591738 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:48.591712 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9e25e2_d171_4142_82a6_a877a50cbea6.slice/crio-e0ea7cb0bb2670ec1de2e500bd4d3003d5f5e587553591ba1cbc62fab2582c28 WatchSource:0}: Error finding container e0ea7cb0bb2670ec1de2e500bd4d3003d5f5e587553591ba1cbc62fab2582c28: Status 404 returned error can't find the container with id e0ea7cb0bb2670ec1de2e500bd4d3003d5f5e587553591ba1cbc62fab2582c28 Nov 04 01:29:48.668397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.668371 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7"] Nov 04 01:29:48.670035 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:48.670007 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e65f77_0ff8_420f_94a6_b029fb7916c9.slice/crio-d12a8e4092bf875bb86c906011234ea7254d76e5ba0c05bb0c2811c945866bc8 WatchSource:0}: Error finding container d12a8e4092bf875bb86c906011234ea7254d76e5ba0c05bb0c2811c945866bc8: Status 404 returned error can't find the container with id d12a8e4092bf875bb86c906011234ea7254d76e5ba0c05bb0c2811c945866bc8 Nov 04 01:29:48.796827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.796801 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs"] Nov 04 01:29:48.797462 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:48.797439 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1cbc30_9a61_4274_b2a4_9da61d518423.slice/crio-8ab2ba5f2fca1bd9bc725f85e37ee13c05e9d5fb1d98047d3e41264b307ca7b1 WatchSource:0}: Error finding container 8ab2ba5f2fca1bd9bc725f85e37ee13c05e9d5fb1d98047d3e41264b307ca7b1: Status 404 returned error can't find the container with id 8ab2ba5f2fca1bd9bc725f85e37ee13c05e9d5fb1d98047d3e41264b307ca7b1 Nov 04 01:29:48.800862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.800820 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:48.800964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.800909 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:48.800964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:48.800957 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:49.483038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.483008 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:29:49.525813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.525777 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" event={"ID":"81e65f77-0ff8-420f-94a6-b029fb7916c9","Type":"ContainerStarted","Data":"5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3"} Nov 04 01:29:49.525986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.525869 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:29:49.525986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.525895 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" event={"ID":"81e65f77-0ff8-420f-94a6-b029fb7916c9","Type":"ContainerStarted","Data":"d12a8e4092bf875bb86c906011234ea7254d76e5ba0c05bb0c2811c945866bc8"} Nov 04 01:29:49.527932 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.527900 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" event={"ID":"be1cbc30-9a61-4274-b2a4-9da61d518423","Type":"ContainerStarted","Data":"cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc"} Nov 04 01:29:49.528141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.528123 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" event={"ID":"be1cbc30-9a61-4274-b2a4-9da61d518423","Type":"ContainerStarted","Data":"8ab2ba5f2fca1bd9bc725f85e37ee13c05e9d5fb1d98047d3e41264b307ca7b1"} Nov 04 01:29:49.529957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.529931 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" event={"ID":"0c9e25e2-d171-4142-82a6-a877a50cbea6","Type":"ContainerStarted","Data":"c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44"} Nov 04 01:29:49.530087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.529971 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" event={"ID":"0c9e25e2-d171-4142-82a6-a877a50cbea6","Type":"ContainerStarted","Data":"e0ea7cb0bb2670ec1de2e500bd4d3003d5f5e587553591ba1cbc62fab2582c28"} Nov 04 01:29:49.530202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.530184 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:29:49.601141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.600849 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" podStartSLOduration=2.600830685 podStartE2EDuration="2.600830685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:49.598798433 +0000 UTC m=+1621.621730442" watchObservedRunningTime="2026-11-04 01:29:49.600830685 +0000 UTC m=+1621.623762693" Nov 04 01:29:49.604383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.604317 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" podStartSLOduration=2.604305314 podStartE2EDuration="2.604305314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:49.577876012 +0000 UTC m=+1621.600808059" watchObservedRunningTime="2026-11-04 01:29:49.604305314 +0000 UTC m=+1621.627237321" Nov 04 01:29:49.623417 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.623302 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podStartSLOduration=1.623287351 podStartE2EDuration="1.623287351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:49.621602205 +0000 UTC m=+1621.644534235" watchObservedRunningTime="2026-11-04 01:29:49.623287351 +0000 UTC m=+1621.646219397" Nov 04 01:29:49.772408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:49.762431 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" podUID="461d2a93-be30-4606-932d-bc5c95097a64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:29:50.074138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:50.074101 4312 scope.go:122] "RemoveContainer" containerID="664b7d9fa9107a68ad4312302e725caf5556217cdf587c6861df8ef08d84bbd1" Nov 04 01:29:50.511373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:50.511267 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:50.513377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:50.513351 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:29:50.513491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:50.513402 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:29:51.023480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:51.023439 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:29:51.023663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:51.023510 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:51.511221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:51.511191 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:29:51.511448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:51.511248 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:29:52.512658 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.511703 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:29:52.512658 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.512081 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:29:52.582716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.582683 4312 generic.go:356] "Generic (PLEG): container finished" podID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerID="6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b" exitCode=0 Nov 04 01:29:52.582887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.582758 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" event={"ID":"d807e2eb-432b-4219-814d-d5bbd21a4c53","Type":"ContainerDied","Data":"6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b"} Nov 04 01:29:52.583228 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.583212 4312 scope.go:122] "RemoveContainer" containerID="6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b" Nov 04 01:29:52.584902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.584872 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:52.584971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.584940 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:52.585011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:52.584976 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:29:53.511522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:53.511476 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:29:53.511700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:53.511539 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:29:53.592412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:53.592379 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" event={"ID":"d807e2eb-432b-4219-814d-d5bbd21a4c53","Type":"ContainerStarted","Data":"cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26"} Nov 04 01:29:54.312440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:54.312402 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:29:54.312627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:54.312480 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:29:54.312627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:54.312532 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:29:54.511289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:54.511257 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:29:54.511571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:54.511549 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:29:55.030391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.030359 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk"] Nov 04 01:29:55.035088 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.035065 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.043418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.043388 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2aefc5-serving-cert\"" Nov 04 01:29:55.047083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.045975 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk"] Nov 04 01:29:55.134277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.134240 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.134479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.134394 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-metrics-certs\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.134479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.134430 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzxl\" (UniqueName: \"kubernetes.io/projected/44ea9217-3aeb-4591-a754-425c549c29f8-kube-api-access-6pzxl\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.134723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.134700 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ea9217-3aeb-4591-a754-425c549c29f8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.236445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.236406 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ea9217-3aeb-4591-a754-425c549c29f8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.236669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.236501 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.236669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.236556 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-metrics-certs\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.236669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.236575 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzxl\" (UniqueName: \"kubernetes.io/projected/44ea9217-3aeb-4591-a754-425c549c29f8-kube-api-access-6pzxl\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.236669 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:55.236661 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-serving-cert: secret "maas-api-e2e-inf-b-2aefc5-serving-cert" not found Nov 04 01:29:55.236893 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:55.236722 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls podName:44ea9217-3aeb-4591-a754-425c549c29f8 nodeName:}" failed. No retries permitted until 2026-11-04 01:29:55.736706343 +0000 UTC m=+1627.759638330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls") pod "maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" (UID: "44ea9217-3aeb-4591-a754-425c549c29f8") : secret "maas-api-e2e-inf-b-2aefc5-serving-cert" not found Nov 04 01:29:55.237076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.237056 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ea9217-3aeb-4591-a754-425c549c29f8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.239134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.239115 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-metrics-certs\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.276162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.276132 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzxl\" (UniqueName: \"kubernetes.io/projected/44ea9217-3aeb-4591-a754-425c549c29f8-kube-api-access-6pzxl\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.312303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.312222 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.312303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.312273 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:29:55.515811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.515778 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:55.516077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.516046 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:55.516756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.516740 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:29:55.745229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.745149 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.756354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.747601 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:55.964539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:55.964499 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:56.304280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:56.304252 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk"] Nov 04 01:29:56.305398 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:29:56.305373 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ea9217_3aeb_4591_a754_425c549c29f8.slice/crio-f587996328ccbe8c34f2b5b63d5a9a37b1af0236faaad85d9b8a8f3f5af7a0b7 WatchSource:0}: Error finding container f587996328ccbe8c34f2b5b63d5a9a37b1af0236faaad85d9b8a8f3f5af7a0b7: Status 404 returned error can't find the container with id f587996328ccbe8c34f2b5b63d5a9a37b1af0236faaad85d9b8a8f3f5af7a0b7 Nov 04 01:29:56.311841 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:56.311818 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:29:56.311932 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:56.311858 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:29:56.644049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:56.644014 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" event={"ID":"44ea9217-3aeb-4591-a754-425c549c29f8","Type":"ContainerStarted","Data":"f587996328ccbe8c34f2b5b63d5a9a37b1af0236faaad85d9b8a8f3f5af7a0b7"} Nov 04 01:29:57.315359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:57.311760 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.315359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:57.311851 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:29:57.666415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:57.665472 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" event={"ID":"44ea9217-3aeb-4591-a754-425c549c29f8","Type":"ContainerStarted","Data":"eb8664ddf8ce83c894011765e0df3b5620d966cfd31ff4e11711189345984b2b"} Nov 04 01:29:57.674355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:57.666741 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:29:57.723484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:57.723408 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" podStartSLOduration=2.239549865 podStartE2EDuration="2.723386623s" totalImagesPullingTime="483.836758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:57.71628453 +0000 UTC m=+1629.739216567" watchObservedRunningTime="2026-11-04 01:29:57.723386623 +0000 UTC m=+1629.746318632" Nov 04 01:29:58.314357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:58.313668 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:29:58.314357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:58.313725 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:29:59.312492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:59.312452 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:29:59.312982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:59.312526 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:29:59.496365 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:29:59.491947 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:29:59.772783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:29:59.772682 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" podUID="461d2a93-be30-4606-932d-bc5c95097a64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:30:00.194970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.194927 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw"] Nov 04 01:30:00.200691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.200666 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.204800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.204771 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw"] Nov 04 01:30:00.272760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.272724 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:30:00.296704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.296674 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b5cb2-ef01-4d87-b48a-878b3239c404-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.296863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.296785 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d97r\" (UniqueName: \"kubernetes.io/projected/fc2b5cb2-ef01-4d87-b48a-878b3239c404-kube-api-access-4d97r\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.316177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.316150 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:30:00.316645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.316595 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:30:00.317329 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.317311 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:30:00.398769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.398721 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b5cb2-ef01-4d87-b48a-878b3239c404-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.399903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.399876 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b5cb2-ef01-4d87-b48a-878b3239c404-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.403538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.403509 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d97r\" (UniqueName: \"kubernetes.io/projected/fc2b5cb2-ef01-4d87-b48a-878b3239c404-kube-api-access-4d97r\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.416033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.416009 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d97r\" (UniqueName: \"kubernetes.io/projected/fc2b5cb2-ef01-4d87-b48a-878b3239c404-kube-api-access-4d97r\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.524849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.524819 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:00.578359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.578115 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:30:00.579913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.579893 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:30:00.718430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.718408 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:30:00.719496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.719440 4312 generic.go:356] "Generic (PLEG): container finished" podID="461d2a93-be30-4606-932d-bc5c95097a64" containerID="ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173" exitCode=0 Nov 04 01:30:00.720137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.720117 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" event={"ID":"461d2a93-be30-4606-932d-bc5c95097a64","Type":"ContainerDied","Data":"ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173"} Nov 04 01:30:00.720196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.720184 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" event={"ID":"461d2a93-be30-4606-932d-bc5c95097a64","Type":"ContainerDied","Data":"40eb9278afd7674279880bff9988c0df14b0b09a97e02dc06c587caead5f590a"} Nov 04 01:30:00.720233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.720208 4312 scope.go:122] "RemoveContainer" containerID="ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173" Nov 04 01:30:00.734500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.734473 4312 scope.go:122] "RemoveContainer" containerID="ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173" Nov 04 01:30:00.735137 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:00.735112 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173\": container with ID starting with ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173 not found: ID does not exist" containerID="ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173" Nov 04 01:30:00.735285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.735258 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173"} err="failed to get container status \"ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173\": rpc error: code = NotFound desc = could not find container \"ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173\": container with ID starting with ee80d667ae676dce77d027e802b770efdb052d0c250286d0160f9c8884d1f173 not found: ID does not exist" Nov 04 01:30:00.810324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.809940 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461d2a93-be30-4606-932d-bc5c95097a64-kube-api-access-mjb9x\" (UniqueName: \"kubernetes.io/projected/461d2a93-be30-4606-932d-bc5c95097a64-kube-api-access-mjb9x\") pod \"461d2a93-be30-4606-932d-bc5c95097a64\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " Nov 04 01:30:00.810324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.810255 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461d2a93-be30-4606-932d-bc5c95097a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461d2a93-be30-4606-932d-bc5c95097a64-plugins-config-volume\") pod \"461d2a93-be30-4606-932d-bc5c95097a64\" (UID: \"461d2a93-be30-4606-932d-bc5c95097a64\") " Nov 04 01:30:00.811435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.811409 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461d2a93-be30-4606-932d-bc5c95097a64-plugins-config-volume" pod "461d2a93-be30-4606-932d-bc5c95097a64" (UID: "461d2a93-be30-4606-932d-bc5c95097a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:00.813169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.813149 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461d2a93-be30-4606-932d-bc5c95097a64-kube-api-access-mjb9x" pod "461d2a93-be30-4606-932d-bc5c95097a64" (UID: "461d2a93-be30-4606-932d-bc5c95097a64"). InnerVolumeSpecName "kube-api-access-mjb9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:00.845035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.845002 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw"] Nov 04 01:30:00.845762 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:30:00.845732 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2b5cb2_ef01_4d87_b48a_878b3239c404.slice/crio-67ed7288953c71c902a7406c484e8d3fd27805fabcb774a80933862ecedfdb31 WatchSource:0}: Error finding container 67ed7288953c71c902a7406c484e8d3fd27805fabcb774a80933862ecedfdb31: Status 404 returned error can't find the container with id 67ed7288953c71c902a7406c484e8d3fd27805fabcb774a80933862ecedfdb31 Nov 04 01:30:00.911972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.911947 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjb9x\" (UniqueName: \"kubernetes.io/projected/461d2a93-be30-4606-932d-bc5c95097a64-kube-api-access-mjb9x\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:00.912090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:00.911977 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461d2a93-be30-4606-932d-bc5c95097a64-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.727758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:01.727662 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" event={"ID":"fc2b5cb2-ef01-4d87-b48a-878b3239c404","Type":"ContainerStarted","Data":"e2bd8bda6099c582c18bbdf2f68c92f332b45d61112e4a25b7999c6377c94b68"} Nov 04 01:30:01.727758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:01.727723 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" event={"ID":"fc2b5cb2-ef01-4d87-b48a-878b3239c404","Type":"ContainerStarted","Data":"67ed7288953c71c902a7406c484e8d3fd27805fabcb774a80933862ecedfdb31"} Nov 04 01:30:01.728751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:01.728727 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2" Nov 04 01:30:01.766649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:01.764732 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" podStartSLOduration=1.7647115420000001 podStartE2EDuration="1.764711542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.74748185 +0000 UTC m=+1633.770413858" watchObservedRunningTime="2026-11-04 01:30:01.764711542 +0000 UTC m=+1633.787643550" Nov 04 01:30:01.781200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:01.781163 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2"] Nov 04 01:30:01.786891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:01.786860 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a5115a-c6449d455-s6qq2"] Nov 04 01:30:02.109598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:02.109561 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461d2a93-be30-4606-932d-bc5c95097a64" path="/var/lib/kubelet/pods/461d2a93-be30-4606-932d-bc5c95097a64/volumes" Nov 04 01:30:02.742736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:02.742703 4312 generic.go:356] "Generic (PLEG): container finished" podID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerID="e2bd8bda6099c582c18bbdf2f68c92f332b45d61112e4a25b7999c6377c94b68" exitCode=60 Nov 04 01:30:02.743195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:02.742743 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" event={"ID":"fc2b5cb2-ef01-4d87-b48a-878b3239c404","Type":"ContainerDied","Data":"e2bd8bda6099c582c18bbdf2f68c92f332b45d61112e4a25b7999c6377c94b68"} Nov 04 01:30:02.743195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:02.743153 4312 scope.go:122] "RemoveContainer" containerID="e2bd8bda6099c582c18bbdf2f68c92f332b45d61112e4a25b7999c6377c94b68" Nov 04 01:30:03.752732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:03.752699 4312 generic.go:356] "Generic (PLEG): container finished" podID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerID="bbca7ee61ee7e8f4e90ad13ccb7135e29949b44a2a5d2d883f5b0070f183ca66" exitCode=60 Nov 04 01:30:03.753226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:03.752779 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" event={"ID":"fc2b5cb2-ef01-4d87-b48a-878b3239c404","Type":"ContainerDied","Data":"bbca7ee61ee7e8f4e90ad13ccb7135e29949b44a2a5d2d883f5b0070f183ca66"} Nov 04 01:30:03.753226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:03.752839 4312 scope.go:122] "RemoveContainer" containerID="e2bd8bda6099c582c18bbdf2f68c92f332b45d61112e4a25b7999c6377c94b68" Nov 04 01:30:03.753226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:03.753161 4312 scope.go:122] "RemoveContainer" containerID="bbca7ee61ee7e8f4e90ad13ccb7135e29949b44a2a5d2d883f5b0070f183ca66" Nov 04 01:30:03.753601 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:03.753578 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw_odh-ai-gateway-infra(fc2b5cb2-ef01-4d87-b48a-878b3239c404)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" Nov 04 01:30:04.187514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:04.187465 4312 patch_prober.go:29] interesting pod/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:30:04.187707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:04.187533 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:04.716160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:04.716130 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:30:04.762827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:04.762797 4312 scope.go:122] "RemoveContainer" containerID="bbca7ee61ee7e8f4e90ad13ccb7135e29949b44a2a5d2d883f5b0070f183ca66" Nov 04 01:30:04.763277 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:04.763195 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw_odh-ai-gateway-infra(fc2b5cb2-ef01-4d87-b48a-878b3239c404)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" Nov 04 01:30:07.764085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.764047 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw"] Nov 04 01:30:07.764684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.764654 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" containerID="cri-o://05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6" gracePeriod=30 Nov 04 01:30:07.777001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.776968 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl"] Nov 04 01:30:07.777231 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.777206 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" containerID="cri-o://45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030" gracePeriod=30 Nov 04 01:30:07.786289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.786262 4312 generic.go:356] "Generic (PLEG): container finished" podID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerID="771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280" exitCode=0 Nov 04 01:30:07.786443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.786360 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" event={"ID":"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4","Type":"ContainerDied","Data":"771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280"} Nov 04 01:30:07.786782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.786767 4312 scope.go:122] "RemoveContainer" containerID="771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280" Nov 04 01:30:07.788404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.788374 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:30:07.788515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.788443 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:30:07.788515 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:07.788473 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:30:08.125465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:08.125427 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:08.125661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:08.125490 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:08.804900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:08.804859 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" event={"ID":"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4","Type":"ContainerStarted","Data":"242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1"} Nov 04 01:30:09.715406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:09.715379 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:30:09.715580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:09.715526 4312 patch_prober.go:29] interesting pod/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:30:09.715634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:09.715577 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:30:10.715444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:10.715412 4312 patch_prober.go:29] interesting pod/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:30:10.715843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:10.715477 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:30:10.825330 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:10.825291 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:10.825521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:10.825371 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:11.715617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:11.715577 4312 patch_prober.go:29] interesting pod/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:30:11.716025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:11.715637 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:30:12.715717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:12.715685 4312 patch_prober.go:29] interesting pod/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:30:12.716145 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:12.715748 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:30:13.101150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.100827 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:30:13.185201 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.185175 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:30:13.231712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231637 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-credential-socket\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.231712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231694 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-socket\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.231927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231738 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-podinfo\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.231927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231830 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-kube-api-access-rshvt\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-kube-api-access-rshvt\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.231927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231866 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-certs\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.232090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231926 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-credential-socket" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.232090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231948 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-envoy\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.232090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.231991 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-data\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.232090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232018 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-socket" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.232297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232123 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4847a0b-f8fa-46e8-873c-76e99b10588c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4847a0b-f8fa-46e8-873c-76e99b10588c-istiod-ca-cert\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.232297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232152 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-token\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-token\") pod \"b4847a0b-f8fa-46e8-873c-76e99b10588c\" (UID: \"b4847a0b-f8fa-46e8-873c-76e99b10588c\") " Nov 04 01:30:13.232297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232161 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-certs" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.232503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232400 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-data" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.232586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232563 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4847a0b-f8fa-46e8-873c-76e99b10588c-istiod-ca-cert" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:13.232850 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232826 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.232948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232856 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.232948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232871 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.232948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232880 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.232948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.232888 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4847a0b-f8fa-46e8-873c-76e99b10588c-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.234664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.234639 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-envoy" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.234766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.234700 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-podinfo" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:13.234766 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.234717 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-kube-api-access-rshvt" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "kube-api-access-rshvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:13.234882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.234867 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-token" pod "b4847a0b-f8fa-46e8-873c-76e99b10588c" (UID: "b4847a0b-f8fa-46e8-873c-76e99b10588c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:13.334081 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334047 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-credential-socket\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334089 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-token\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-token\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334118 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-socket\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334139 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-envoy\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334163 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-podinfo\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334202 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-kube-api-access-dx59g\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-kube-api-access-dx59g\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334254 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-certs\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334388 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3359d44-f1ed-463c-a61e-52f2ce59795d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3359d44-f1ed-463c-a61e-52f2ce59795d-istiod-ca-cert\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334387 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-credential-socket" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.334675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334436 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-data\") pod \"a3359d44-f1ed-463c-a61e-52f2ce59795d\" (UID: \"a3359d44-f1ed-463c-a61e-52f2ce59795d\") " Nov 04 01:30:13.334675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334565 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-socket" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.334902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334711 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-certs" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.334959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334902 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-data" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.335009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.334960 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3359d44-f1ed-463c-a61e-52f2ce59795d-istiod-ca-cert" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:13.335107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335084 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335114 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3359d44-f1ed-463c-a61e-52f2ce59795d-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335128 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335141 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshvt\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-kube-api-access-rshvt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335156 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335169 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335182 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335193 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4847a0b-f8fa-46e8-873c-76e99b10588c-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.335298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.335204 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.337115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.337086 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-token" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:13.337115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.337104 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-podinfo" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:13.337260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.337114 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-envoy" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:13.337260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.337118 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-kube-api-access-dx59g" pod "a3359d44-f1ed-463c-a61e-52f2ce59795d" (UID: "a3359d44-f1ed-463c-a61e-52f2ce59795d"). InnerVolumeSpecName "kube-api-access-dx59g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:13.436472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.436375 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.436472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.436411 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.436472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.436426 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3359d44-f1ed-463c-a61e-52f2ce59795d-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.436472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.436442 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx59g\" (UniqueName: \"kubernetes.io/projected/a3359d44-f1ed-463c-a61e-52f2ce59795d-kube-api-access-dx59g\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:13.714877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.714841 4312 patch_prober.go:29] interesting pod/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:30:13.715059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.714900 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:30:13.855420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.855385 4312 generic.go:356] "Generic (PLEG): container finished" podID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerID="45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030" exitCode=0 Nov 04 01:30:13.855866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.855435 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" Nov 04 01:30:13.855866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.855473 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" event={"ID":"a3359d44-f1ed-463c-a61e-52f2ce59795d","Type":"ContainerDied","Data":"45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030"} Nov 04 01:30:13.855866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.855571 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl" event={"ID":"a3359d44-f1ed-463c-a61e-52f2ce59795d","Type":"ContainerDied","Data":"17db6f2485a924c882faac7e488008e5e4c51b2640cb1b0c55ca0c936b448072"} Nov 04 01:30:13.855866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.855614 4312 scope.go:122] "RemoveContainer" containerID="45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030" Nov 04 01:30:13.857259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.857239 4312 generic.go:356] "Generic (PLEG): container finished" podID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerID="05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6" exitCode=0 Nov 04 01:30:13.857390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.857285 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" event={"ID":"b4847a0b-f8fa-46e8-873c-76e99b10588c","Type":"ContainerDied","Data":"05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6"} Nov 04 01:30:13.857390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.857308 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" Nov 04 01:30:13.857390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.857317 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw" event={"ID":"b4847a0b-f8fa-46e8-873c-76e99b10588c","Type":"ContainerDied","Data":"dd677a415ea73be3e9b74f0341154af9ee10ab98cc81c8af7a1f5db2c00f7425"} Nov 04 01:30:13.866923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.866848 4312 scope.go:122] "RemoveContainer" containerID="aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8" Nov 04 01:30:13.876485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.876465 4312 scope.go:122] "RemoveContainer" containerID="45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030" Nov 04 01:30:13.876788 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:13.876766 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030\": container with ID starting with 45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030 not found: ID does not exist" containerID="45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030" Nov 04 01:30:13.876875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.876793 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030"} err="failed to get container status \"45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030\": rpc error: code = NotFound desc = could not find container \"45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030\": container with ID starting with 45ae2cfc30f6cfe929b5a06d3eb1a78b54ecd8978fe6276aaca381a420f03030 not found: ID does not exist" Nov 04 01:30:13.876875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.876808 4312 scope.go:122] "RemoveContainer" containerID="aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8" Nov 04 01:30:13.877063 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:13.877046 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8\": container with ID starting with aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8 not found: ID does not exist" containerID="aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8" Nov 04 01:30:13.877108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.877069 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8"} err="failed to get container status \"aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8\": rpc error: code = NotFound desc = could not find container \"aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8\": container with ID starting with aa0ab7a4f55f8d9261d77e9565480f5c2ca4a86323fd6a0dd647b73db4f372d8 not found: ID does not exist" Nov 04 01:30:13.877108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.877091 4312 scope.go:122] "RemoveContainer" containerID="05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6" Nov 04 01:30:13.885955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.885934 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl"] Nov 04 01:30:13.887477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.887418 4312 scope.go:122] "RemoveContainer" containerID="05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6" Nov 04 01:30:13.888141 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:13.888115 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6\": container with ID starting with 05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6 not found: ID does not exist" containerID="05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6" Nov 04 01:30:13.888219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.888152 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6"} err="failed to get container status \"05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6\": rpc error: code = NotFound desc = could not find container \"05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6\": container with ID starting with 05b86171844e445bf73d694c17ed0cc91b8897aeefcd87730f3d1604d0acd7f6 not found: ID does not exist" Nov 04 01:30:13.891394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.891376 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-qdfnl"] Nov 04 01:30:13.905231 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.905209 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw"] Nov 04 01:30:13.917602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:13.917573 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-ddfgw"] Nov 04 01:30:14.125235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:14.125204 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" path="/var/lib/kubelet/pods/a3359d44-f1ed-463c-a61e-52f2ce59795d/volumes" Nov 04 01:30:14.125897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:14.125875 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" path="/var/lib/kubelet/pods/b4847a0b-f8fa-46e8-873c-76e99b10588c/volumes" Nov 04 01:30:14.719295 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:14.719266 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:30:14.719754 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:14.719735 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:30:14.720430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:14.720411 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:30:15.083985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.083953 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk"] Nov 04 01:30:15.084816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084795 4312 cpu_manager.go:414] "containerMap: removing container" podUID="461d2a93-be30-4606-932d-bc5c95097a64" containerName="payload-pre-processing" Nov 04 01:30:15.084926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084818 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461d2a93-be30-4606-932d-bc5c95097a64" containerName="payload-pre-processing" Nov 04 01:30:15.084926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084846 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" Nov 04 01:30:15.084926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084854 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" Nov 04 01:30:15.084926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084870 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" Nov 04 01:30:15.084926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084877 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" Nov 04 01:30:15.084926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084897 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" Nov 04 01:30:15.084926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.084903 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" Nov 04 01:30:15.085273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.085081 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="461d2a93-be30-4606-932d-bc5c95097a64" containerName="payload-pre-processing" Nov 04 01:30:15.085273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.085099 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" Nov 04 01:30:15.085273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.085109 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4847a0b-f8fa-46e8-873c-76e99b10588c" containerName="istio-proxy" Nov 04 01:30:15.085273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.085120 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3359d44-f1ed-463c-a61e-52f2ce59795d" containerName="istio-proxy" Nov 04 01:30:15.090571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.090546 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.098617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.098586 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk"] Nov 04 01:30:15.098732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.098629 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8974fa\"/\"openshift-service-ca.crt\"" Nov 04 01:30:15.099556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.099534 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8974fa\"/\"kube-root-ca.crt\"" Nov 04 01:30:15.116842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.116810 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8974fa\"/\"default-dockercfg-pdnnd\"" Nov 04 01:30:15.116978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.116868 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8974fa\"/\"test-model-8974fa-kserve-self-signed-certs\"" Nov 04 01:30:15.159192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.159163 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-model-cache\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.159313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.159209 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tmp-dir\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.159313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.159292 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkls\" (UniqueName: \"kubernetes.io/projected/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-kube-api-access-fwkls\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.159444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.159429 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-dshm\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.159479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.159459 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-home\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.159526 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.159511 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tls-certs\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.260557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.260524 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tls-certs\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.260725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.260636 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-model-cache\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.260725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.260670 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tmp-dir\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.260725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.260708 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkls\" (UniqueName: \"kubernetes.io/projected/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-kube-api-access-fwkls\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.260913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.260776 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-dshm\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.260913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.260797 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-home\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.261107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.261084 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-model-cache\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.261189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.261103 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tmp-dir\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.261225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.261196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-home\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.262943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.262918 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-dshm\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.263062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.263043 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tls-certs\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.306160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.306114 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkls\" (UniqueName: \"kubernetes.io/projected/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-kube-api-access-fwkls\") pod \"test-model-8974fa-kserve-6c9cfcdf7c-4fbfk\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.412607 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.412526 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:15.696097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.696060 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk"] Nov 04 01:30:15.696859 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:30:15.696834 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5d2a781_cc39_40f3_b2fe_cd5de79d89ff.slice/crio-77d11e2da84cf70d42c738023191248f2a4a9f7fedf6300048379bc507c763eb WatchSource:0}: Error finding container 77d11e2da84cf70d42c738023191248f2a4a9f7fedf6300048379bc507c763eb: Status 404 returned error can't find the container with id 77d11e2da84cf70d42c738023191248f2a4a9f7fedf6300048379bc507c763eb Nov 04 01:30:15.877541 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.877495 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" event={"ID":"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff","Type":"ContainerStarted","Data":"f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19"} Nov 04 01:30:15.877750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.877557 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" event={"ID":"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff","Type":"ContainerStarted","Data":"77d11e2da84cf70d42c738023191248f2a4a9f7fedf6300048379bc507c763eb"} Nov 04 01:30:15.898883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:15.898829 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" podStartSLOduration=0.898812515 podStartE2EDuration="898.812515ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:15.896367968 +0000 UTC m=+1647.919300000" watchObservedRunningTime="2026-11-04 01:30:15.898812515 +0000 UTC m=+1647.921744522" Nov 04 01:30:16.100895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:16.100866 4312 scope.go:122] "RemoveContainer" containerID="bbca7ee61ee7e8f4e90ad13ccb7135e29949b44a2a5d2d883f5b0070f183ca66" Nov 04 01:30:16.413399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:16.413372 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:16.887894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:16.887851 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" event={"ID":"fc2b5cb2-ef01-4d87-b48a-878b3239c404","Type":"ContainerStarted","Data":"4a4e0d8f7587e24114b925d2895b5ba2eee5fa91fbe352f02592a1ba7a5b9c47"} Nov 04 01:30:17.897005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:17.896971 4312 generic.go:356] "Generic (PLEG): container finished" podID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerID="4a4e0d8f7587e24114b925d2895b5ba2eee5fa91fbe352f02592a1ba7a5b9c47" exitCode=60 Nov 04 01:30:17.897441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:17.897040 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" event={"ID":"fc2b5cb2-ef01-4d87-b48a-878b3239c404","Type":"ContainerDied","Data":"4a4e0d8f7587e24114b925d2895b5ba2eee5fa91fbe352f02592a1ba7a5b9c47"} Nov 04 01:30:17.897441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:17.897098 4312 scope.go:122] "RemoveContainer" containerID="bbca7ee61ee7e8f4e90ad13ccb7135e29949b44a2a5d2d883f5b0070f183ca66" Nov 04 01:30:17.897682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:17.897665 4312 scope.go:122] "RemoveContainer" containerID="4a4e0d8f7587e24114b925d2895b5ba2eee5fa91fbe352f02592a1ba7a5b9c47" Nov 04 01:30:17.897970 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:17.897952 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw_odh-ai-gateway-infra(fc2b5cb2-ef01-4d87-b48a-878b3239c404)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" Nov 04 01:30:17.916738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:17.916716 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw"] Nov 04 01:30:19.073968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.073942 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:19.228998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.228922 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc2b5cb2-ef01-4d87-b48a-878b3239c404-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b5cb2-ef01-4d87-b48a-878b3239c404-maas-api-service-ca\") pod \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " Nov 04 01:30:19.228998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.228959 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2b5cb2-ef01-4d87-b48a-878b3239c404-kube-api-access-4d97r\" (UniqueName: \"kubernetes.io/projected/fc2b5cb2-ef01-4d87-b48a-878b3239c404-kube-api-access-4d97r\") pod \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\" (UID: \"fc2b5cb2-ef01-4d87-b48a-878b3239c404\") " Nov 04 01:30:19.229688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.229355 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2b5cb2-ef01-4d87-b48a-878b3239c404-maas-api-service-ca" pod "fc2b5cb2-ef01-4d87-b48a-878b3239c404" (UID: "fc2b5cb2-ef01-4d87-b48a-878b3239c404"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:19.231578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.231554 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2b5cb2-ef01-4d87-b48a-878b3239c404-kube-api-access-4d97r" pod "fc2b5cb2-ef01-4d87-b48a-878b3239c404" (UID: "fc2b5cb2-ef01-4d87-b48a-878b3239c404"). InnerVolumeSpecName "kube-api-access-4d97r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:19.330620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.330593 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b5cb2-ef01-4d87-b48a-878b3239c404-maas-api-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:19.330620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.330617 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d97r\" (UniqueName: \"kubernetes.io/projected/fc2b5cb2-ef01-4d87-b48a-878b3239c404-kube-api-access-4d97r\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:19.914092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.914049 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" Nov 04 01:30:19.914092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.914086 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw" event={"ID":"fc2b5cb2-ef01-4d87-b48a-878b3239c404","Type":"ContainerDied","Data":"67ed7288953c71c902a7406c484e8d3fd27805fabcb774a80933862ecedfdb31"} Nov 04 01:30:19.914346 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.914146 4312 scope.go:122] "RemoveContainer" containerID="4a4e0d8f7587e24114b925d2895b5ba2eee5fa91fbe352f02592a1ba7a5b9c47" Nov 04 01:30:19.943770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.943740 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw"] Nov 04 01:30:19.950429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:19.950403 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895930-x5qhw"] Nov 04 01:30:20.111187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:20.111150 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" path="/var/lib/kubelet/pods/fc2b5cb2-ef01-4d87-b48a-878b3239c404/volumes" Nov 04 01:30:25.412639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:25.412605 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:25.425428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:25.425398 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:25.965659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:30:25.965628 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:30:29.458598 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:29.458511 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:30:59.457510 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:30:59.457469 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:31:27.525435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:27.525394 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v"] Nov 04 01:31:27.526079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:27.525658 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" containerID="cri-o://5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f" gracePeriod=30 Nov 04 01:31:29.456865 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:31:29.456829 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:31:32.791408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.791382 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:31:32.937219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937136 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-credential-socket\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937197 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-socket\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937244 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-envoy\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937311 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-certs\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937427 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-credential-socket" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:32.937629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937439 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-data\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937537 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b46565c-06b6-43ea-8d37-da745354cad9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46565c-06b6-43ea-8d37-da745354cad9-istio-podinfo\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937592 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-socket" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:32.937774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937637 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-data" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:32.937774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937665 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-certs" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:32.937774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937700 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-istio-token\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-istio-token\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937750 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-kube-api-access-rj4c5\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-kube-api-access-rj4c5\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.937774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.937771 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b46565c-06b6-43ea-8d37-da745354cad9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46565c-06b6-43ea-8d37-da745354cad9-istiod-ca-cert\") pod \"1b46565c-06b6-43ea-8d37-da745354cad9\" (UID: \"1b46565c-06b6-43ea-8d37-da745354cad9\") " Nov 04 01:31:32.938182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.938156 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b46565c-06b6-43ea-8d37-da745354cad9-istiod-ca-cert" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:32.938411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.938392 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:32.938505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.938413 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:32.938505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.938422 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:32.938505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.938432 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:32.938505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.938440 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46565c-06b6-43ea-8d37-da745354cad9-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:32.939612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.939581 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-envoy" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:32.939731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.939672 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b46565c-06b6-43ea-8d37-da745354cad9-istio-podinfo" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:32.940192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.940173 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-kube-api-access-rj4c5" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "kube-api-access-rj4c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:32.940278 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:32.940198 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-istio-token" pod "1b46565c-06b6-43ea-8d37-da745354cad9" (UID: "1b46565c-06b6-43ea-8d37-da745354cad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:33.039219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.039191 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46565c-06b6-43ea-8d37-da745354cad9-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.039219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.039216 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.039439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.039227 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj4c5\" (UniqueName: \"kubernetes.io/projected/1b46565c-06b6-43ea-8d37-da745354cad9-kube-api-access-rj4c5\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.039439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.039237 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46565c-06b6-43ea-8d37-da745354cad9-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.440040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.440004 4312 generic.go:356] "Generic (PLEG): container finished" podID="1b46565c-06b6-43ea-8d37-da745354cad9" containerID="5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f" exitCode=0 Nov 04 01:31:33.440211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.440053 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" Nov 04 01:31:33.440211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.440079 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" event={"ID":"1b46565c-06b6-43ea-8d37-da745354cad9","Type":"ContainerDied","Data":"5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f"} Nov 04 01:31:33.440211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.440142 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v" event={"ID":"1b46565c-06b6-43ea-8d37-da745354cad9","Type":"ContainerDied","Data":"cf3a590c0c7a33443c3d3c45ee9ba4ab90cbecd42504fa01baab9ed9fec81690"} Nov 04 01:31:33.440211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.440165 4312 scope.go:122] "RemoveContainer" containerID="5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f" Nov 04 01:31:33.451244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.451225 4312 scope.go:122] "RemoveContainer" containerID="5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f" Nov 04 01:31:33.451599 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:31:33.451567 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f\": container with ID starting with 5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f not found: ID does not exist" containerID="5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f" Nov 04 01:31:33.451703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.451608 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f"} err="failed to get container status \"5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f\": rpc error: code = NotFound desc = could not find container \"5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f\": container with ID starting with 5e6b2f8179219d9602fa91076a93d4fb7a6215157f2f163f2f384bd0b6ef8a9f not found: ID does not exist" Nov 04 01:31:33.464867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.464844 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v"] Nov 04 01:31:33.468982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:33.468957 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-2r49v"] Nov 04 01:31:34.112123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:31:34.112087 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" path="/var/lib/kubelet/pods/1b46565c-06b6-43ea-8d37-da745354cad9/volumes" Nov 04 01:31:59.461785 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:31:59.458311 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:32:18.365067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:18.365032 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z"] Nov 04 01:32:18.381683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:18.381651 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq"] Nov 04 01:32:21.049281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.049243 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk"] Nov 04 01:32:21.303202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.303123 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x"] Nov 04 01:32:21.303408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.303360 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" podUID="9a85f039-b6f3-4686-9529-115a20f63093" containerName="maas-api" containerID="cri-o://f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa" gracePeriod=30 Nov 04 01:32:21.631500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.631465 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:32:21.717115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.717082 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-metrics-certs\") pod \"9a85f039-b6f3-4686-9529-115a20f63093\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " Nov 04 01:32:21.717306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.717173 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a85f039-b6f3-4686-9529-115a20f63093-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a85f039-b6f3-4686-9529-115a20f63093-openshift-service-ca\") pod \"9a85f039-b6f3-4686-9529-115a20f63093\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " Nov 04 01:32:21.717306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.717213 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a85f039-b6f3-4686-9529-115a20f63093-kube-api-access-xw7xv\" (UniqueName: \"kubernetes.io/projected/9a85f039-b6f3-4686-9529-115a20f63093-kube-api-access-xw7xv\") pod \"9a85f039-b6f3-4686-9529-115a20f63093\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " Nov 04 01:32:21.717306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.717287 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls\") pod \"9a85f039-b6f3-4686-9529-115a20f63093\" (UID: \"9a85f039-b6f3-4686-9529-115a20f63093\") " Nov 04 01:32:21.717627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.717605 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a85f039-b6f3-4686-9529-115a20f63093-openshift-service-ca" pod "9a85f039-b6f3-4686-9529-115a20f63093" (UID: "9a85f039-b6f3-4686-9529-115a20f63093"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:21.717793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.717771 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a85f039-b6f3-4686-9529-115a20f63093-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:21.719250 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.719229 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-metrics-certs" pod "9a85f039-b6f3-4686-9529-115a20f63093" (UID: "9a85f039-b6f3-4686-9529-115a20f63093"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:21.719392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.719370 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a85f039-b6f3-4686-9529-115a20f63093-kube-api-access-xw7xv" pod "9a85f039-b6f3-4686-9529-115a20f63093" (UID: "9a85f039-b6f3-4686-9529-115a20f63093"). InnerVolumeSpecName "kube-api-access-xw7xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:21.719505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.719490 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls" pod "9a85f039-b6f3-4686-9529-115a20f63093" (UID: "9a85f039-b6f3-4686-9529-115a20f63093"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:21.753900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.753875 4312 generic.go:356] "Generic (PLEG): container finished" podID="9a85f039-b6f3-4686-9529-115a20f63093" containerID="f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa" exitCode=0 Nov 04 01:32:21.754030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.753918 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" Nov 04 01:32:21.754030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.753970 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" event={"ID":"9a85f039-b6f3-4686-9529-115a20f63093","Type":"ContainerDied","Data":"f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa"} Nov 04 01:32:21.754030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.754025 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x" event={"ID":"9a85f039-b6f3-4686-9529-115a20f63093","Type":"ContainerDied","Data":"926d8a64162a8853ea7318d9229b1c6fc2cb8d3f32b4cbfc90de9740b5a344f8"} Nov 04 01:32:21.754155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.754053 4312 scope.go:122] "RemoveContainer" containerID="f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa" Nov 04 01:32:21.765303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.765279 4312 scope.go:122] "RemoveContainer" containerID="f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa" Nov 04 01:32:21.765608 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:21.765587 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa\": container with ID starting with f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa not found: ID does not exist" containerID="f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa" Nov 04 01:32:21.765665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.765618 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa"} err="failed to get container status \"f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa\": rpc error: code = NotFound desc = could not find container \"f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa\": container with ID starting with f92038e3997fa0bba3da735c80ac6d6b83864993f8759d5147f985b54b625afa not found: ID does not exist" Nov 04 01:32:21.781000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.780978 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x"] Nov 04 01:32:21.784959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.784937 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a362b6-6484576b98-6nz9x"] Nov 04 01:32:21.822039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.821992 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:21.822234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.822043 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7xv\" (UniqueName: \"kubernetes.io/projected/9a85f039-b6f3-4686-9529-115a20f63093-kube-api-access-xw7xv\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:21.822234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:21.822061 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a85f039-b6f3-4686-9529-115a20f63093-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:22.108827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:22.108797 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a85f039-b6f3-4686-9529-115a20f63093" path="/var/lib/kubelet/pods/9a85f039-b6f3-4686-9529-115a20f63093/volumes" Nov 04 01:32:23.365770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.365730 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" podUID="8733aede-0f22-4ebe-90e5-636a7409a56d" containerName="payload-processing" containerID="cri-o://41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf" gracePeriod=25 Nov 04 01:32:23.382822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.382788 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" podUID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" containerName="payload-pre-processing" containerID="cri-o://fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0" gracePeriod=25 Nov 04 01:32:23.653679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.653656 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:32:23.748010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.747974 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8733aede-0f22-4ebe-90e5-636a7409a56d-kube-api-access-g2mld\" (UniqueName: \"kubernetes.io/projected/8733aede-0f22-4ebe-90e5-636a7409a56d-kube-api-access-g2mld\") pod \"8733aede-0f22-4ebe-90e5-636a7409a56d\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " Nov 04 01:32:23.748196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.748100 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8733aede-0f22-4ebe-90e5-636a7409a56d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8733aede-0f22-4ebe-90e5-636a7409a56d-plugins-config-volume\") pod \"8733aede-0f22-4ebe-90e5-636a7409a56d\" (UID: \"8733aede-0f22-4ebe-90e5-636a7409a56d\") " Nov 04 01:32:23.748480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.748456 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8733aede-0f22-4ebe-90e5-636a7409a56d-plugins-config-volume" pod "8733aede-0f22-4ebe-90e5-636a7409a56d" (UID: "8733aede-0f22-4ebe-90e5-636a7409a56d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:23.750042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.750022 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8733aede-0f22-4ebe-90e5-636a7409a56d-kube-api-access-g2mld" pod "8733aede-0f22-4ebe-90e5-636a7409a56d" (UID: "8733aede-0f22-4ebe-90e5-636a7409a56d"). InnerVolumeSpecName "kube-api-access-g2mld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:23.768844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.768818 4312 generic.go:356] "Generic (PLEG): container finished" podID="8733aede-0f22-4ebe-90e5-636a7409a56d" containerID="41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf" exitCode=0 Nov 04 01:32:23.768971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.768849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" event={"ID":"8733aede-0f22-4ebe-90e5-636a7409a56d","Type":"ContainerDied","Data":"41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf"} Nov 04 01:32:23.768971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.768871 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" Nov 04 01:32:23.768971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.768889 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z" event={"ID":"8733aede-0f22-4ebe-90e5-636a7409a56d","Type":"ContainerDied","Data":"3aec2edf96bcd634344798234a0743c789b7d623dbe956b254c5ee72b5d46bef"} Nov 04 01:32:23.768971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.768911 4312 scope.go:122] "RemoveContainer" containerID="41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf" Nov 04 01:32:23.779013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.778996 4312 scope.go:122] "RemoveContainer" containerID="41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf" Nov 04 01:32:23.779301 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:23.779275 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf\": container with ID starting with 41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf not found: ID does not exist" containerID="41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf" Nov 04 01:32:23.779364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.779312 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf"} err="failed to get container status \"41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf\": rpc error: code = NotFound desc = could not find container \"41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf\": container with ID starting with 41b4dff63738d5ae892eec7b58344723c9a3be768a2c7ed37e558ab87bfe36bf not found: ID does not exist" Nov 04 01:32:23.793427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.793402 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z"] Nov 04 01:32:23.799810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.799792 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a362b6-d4c7b4dfb-xhk2z"] Nov 04 01:32:23.849330 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.849306 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2mld\" (UniqueName: \"kubernetes.io/projected/8733aede-0f22-4ebe-90e5-636a7409a56d-kube-api-access-g2mld\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:23.849466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:23.849329 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8733aede-0f22-4ebe-90e5-636a7409a56d-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:24.109977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:24.109933 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8733aede-0f22-4ebe-90e5-636a7409a56d" path="/var/lib/kubelet/pods/8733aede-0f22-4ebe-90e5-636a7409a56d/volumes" Nov 04 01:32:27.281350 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:27.281296 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr"] Nov 04 01:32:27.281818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:27.281612 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" containerID="cri-o://482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb" gracePeriod=30 Nov 04 01:32:27.834325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:27.834274 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" podUID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:29.458830 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:29.458788 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:32:32.573055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.573031 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:32:32.682677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.682647 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-data\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.682855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.682763 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-envoy\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.682909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.682890 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-token\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-token\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.682957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.682919 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-data" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:32.682957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.682947 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-kube-api-access-nkfdx\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-kube-api-access-nkfdx\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.683045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.682989 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-socket\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.683045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.683020 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-certs\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.683141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.683099 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-podinfo\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.683316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.683282 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-socket" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:32.683434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.683385 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-certs" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:32.683566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.683541 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-credential-socket\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.683775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.683560 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-credential-socket" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:32.683775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.683752 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4204c8e6-9476-4a48-a947-635f16cb21c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4204c8e6-9476-4a48-a947-635f16cb21c7-istiod-ca-cert\") pod \"4204c8e6-9476-4a48-a947-635f16cb21c7\" (UID: \"4204c8e6-9476-4a48-a947-635f16cb21c7\") " Nov 04 01:32:32.684151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.684125 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4204c8e6-9476-4a48-a947-635f16cb21c7-istiod-ca-cert" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:32.684452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.684431 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.684540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.684457 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.684540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.684473 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.684540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.684487 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4204c8e6-9476-4a48-a947-635f16cb21c7-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.684540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.684500 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.685155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.685133 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-envoy" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:32.685382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.685360 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-kube-api-access-nkfdx" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "kube-api-access-nkfdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:32.685464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.685398 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-token" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:32.685568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.685548 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-podinfo" pod "4204c8e6-9476-4a48-a947-635f16cb21c7" (UID: "4204c8e6-9476-4a48-a947-635f16cb21c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:32.785684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.785654 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.785684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.785680 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkfdx\" (UniqueName: \"kubernetes.io/projected/4204c8e6-9476-4a48-a947-635f16cb21c7-kube-api-access-nkfdx\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.785684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.785692 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.785917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.785701 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4204c8e6-9476-4a48-a947-635f16cb21c7-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:32.850349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.850257 4312 generic.go:356] "Generic (PLEG): container finished" podID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerID="482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb" exitCode=0 Nov 04 01:32:32.850349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.850308 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" Nov 04 01:32:32.850537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.850396 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" event={"ID":"4204c8e6-9476-4a48-a947-635f16cb21c7","Type":"ContainerDied","Data":"482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb"} Nov 04 01:32:32.850537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.850450 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr" event={"ID":"4204c8e6-9476-4a48-a947-635f16cb21c7","Type":"ContainerDied","Data":"e71407aa36090cf168276ec90417dcb9ff4b01e42d429e091078a97c782d2b26"} Nov 04 01:32:32.850537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.850472 4312 scope.go:122] "RemoveContainer" containerID="482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb" Nov 04 01:32:32.861746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.861725 4312 scope.go:122] "RemoveContainer" containerID="4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191" Nov 04 01:32:32.870379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.870364 4312 scope.go:122] "RemoveContainer" containerID="482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb" Nov 04 01:32:32.870647 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:32.870623 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb\": container with ID starting with 482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb not found: ID does not exist" containerID="482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb" Nov 04 01:32:32.870735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.870654 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb"} err="failed to get container status \"482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb\": rpc error: code = NotFound desc = could not find container \"482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb\": container with ID starting with 482c576cd2af4fcb0acd9066d8bf07c272860dc42e7d1d64687e37d7cf420beb not found: ID does not exist" Nov 04 01:32:32.870735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.870670 4312 scope.go:122] "RemoveContainer" containerID="4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191" Nov 04 01:32:32.870897 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:32.870879 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191\": container with ID starting with 4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191 not found: ID does not exist" containerID="4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191" Nov 04 01:32:32.870947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.870906 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191"} err="failed to get container status \"4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191\": rpc error: code = NotFound desc = could not find container \"4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191\": container with ID starting with 4a1fc9435d3eca27b9d20e981d1680c535141bd5243f2985fb5cc277e711c191 not found: ID does not exist" Nov 04 01:32:32.876701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.876679 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr"] Nov 04 01:32:32.881855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:32.881836 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-f47zr"] Nov 04 01:32:34.112018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:34.111980 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" path="/var/lib/kubelet/pods/4204c8e6-9476-4a48-a947-635f16cb21c7/volumes" Nov 04 01:32:34.658975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:34.658936 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt"] Nov 04 01:32:34.675323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:34.675290 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb"] Nov 04 01:32:36.070895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.070849 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" podUID="b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" containerName="main" containerID="cri-o://f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19" gracePeriod=45 Nov 04 01:32:36.152452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.152419 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf"] Nov 04 01:32:36.152694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.152669 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" podUID="fd92106e-d547-4052-a6a0-de9222fa375a" containerName="maas-api" containerID="cri-o://79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c" gracePeriod=30 Nov 04 01:32:36.362114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.361727 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:32:36.440425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.440390 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tls-certs\") pod \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " Nov 04 01:32:36.440588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.440489 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-dshm\") pod \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " Nov 04 01:32:36.440588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.440521 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-kube-api-access-fwkls\" (UniqueName: \"kubernetes.io/projected/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-kube-api-access-fwkls\") pod \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " Nov 04 01:32:36.440588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.440577 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-home\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-home\") pod \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " Nov 04 01:32:36.440755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.440664 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-model-cache\") pod \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " Nov 04 01:32:36.440833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.440816 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tmp-dir\") pod \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\" (UID: \"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff\") " Nov 04 01:32:36.441581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.441552 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tmp-dir" pod "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" (UID: "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:36.441775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.441751 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-home" pod "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" (UID: "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:36.441971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.441941 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-model-cache" pod "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" (UID: "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:36.443918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.443888 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-kube-api-access-fwkls" pod "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" (UID: "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff"). InnerVolumeSpecName "kube-api-access-fwkls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:36.444699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.444654 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tls-certs" pod "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" (UID: "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:36.446366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.446299 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-dshm" pod "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" (UID: "b5d2a781-cc39-40f3-b2fe-cd5de79d89ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:36.543424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.543388 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.543424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.543423 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.543671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.543438 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.543671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.543450 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.543671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.543464 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkls\" (UniqueName: \"kubernetes.io/projected/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-kube-api-access-fwkls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.543671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.543479 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.589244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.589220 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:32:36.644164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.644073 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd92106e-d547-4052-a6a0-de9222fa375a-kube-api-access-v95r6\" (UniqueName: \"kubernetes.io/projected/fd92106e-d547-4052-a6a0-de9222fa375a-kube-api-access-v95r6\") pod \"fd92106e-d547-4052-a6a0-de9222fa375a\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " Nov 04 01:32:36.644357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.644233 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd92106e-d547-4052-a6a0-de9222fa375a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd92106e-d547-4052-a6a0-de9222fa375a-openshift-service-ca\") pod \"fd92106e-d547-4052-a6a0-de9222fa375a\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " Nov 04 01:32:36.644357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.644285 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-metrics-certs\") pod \"fd92106e-d547-4052-a6a0-de9222fa375a\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " Nov 04 01:32:36.644489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.644417 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-maas-api-tls\") pod \"fd92106e-d547-4052-a6a0-de9222fa375a\" (UID: \"fd92106e-d547-4052-a6a0-de9222fa375a\") " Nov 04 01:32:36.644684 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.644654 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd92106e-d547-4052-a6a0-de9222fa375a-openshift-service-ca" pod "fd92106e-d547-4052-a6a0-de9222fa375a" (UID: "fd92106e-d547-4052-a6a0-de9222fa375a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:36.645125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.645099 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd92106e-d547-4052-a6a0-de9222fa375a-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.646665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.646631 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-maas-api-tls" pod "fd92106e-d547-4052-a6a0-de9222fa375a" (UID: "fd92106e-d547-4052-a6a0-de9222fa375a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:36.647058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.647034 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd92106e-d547-4052-a6a0-de9222fa375a-kube-api-access-v95r6" pod "fd92106e-d547-4052-a6a0-de9222fa375a" (UID: "fd92106e-d547-4052-a6a0-de9222fa375a"). InnerVolumeSpecName "kube-api-access-v95r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:36.647878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.647855 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-metrics-certs" pod "fd92106e-d547-4052-a6a0-de9222fa375a" (UID: "fd92106e-d547-4052-a6a0-de9222fa375a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:36.746442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.746410 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v95r6\" (UniqueName: \"kubernetes.io/projected/fd92106e-d547-4052-a6a0-de9222fa375a-kube-api-access-v95r6\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.746442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.746438 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.746442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.746448 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd92106e-d547-4052-a6a0-de9222fa375a-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.881249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.881213 4312 generic.go:356] "Generic (PLEG): container finished" podID="fd92106e-d547-4052-a6a0-de9222fa375a" containerID="79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c" exitCode=0 Nov 04 01:32:36.881452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.881269 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" Nov 04 01:32:36.881452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.881302 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" event={"ID":"fd92106e-d547-4052-a6a0-de9222fa375a","Type":"ContainerDied","Data":"79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c"} Nov 04 01:32:36.881452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.881386 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf" event={"ID":"fd92106e-d547-4052-a6a0-de9222fa375a","Type":"ContainerDied","Data":"1a4e141ce9cf8d210b3c4df51a7628ee3c8a639b26622a892f2ed1a3897a6e85"} Nov 04 01:32:36.881452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.881399 4312 scope.go:122] "RemoveContainer" containerID="79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c" Nov 04 01:32:36.882796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.882774 4312 generic.go:356] "Generic (PLEG): container finished" podID="b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" containerID="f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19" exitCode=0 Nov 04 01:32:36.882933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.882816 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" Nov 04 01:32:36.882933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.882857 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" event={"ID":"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff","Type":"ContainerDied","Data":"f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19"} Nov 04 01:32:36.882933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.882890 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk" event={"ID":"b5d2a781-cc39-40f3-b2fe-cd5de79d89ff","Type":"ContainerDied","Data":"77d11e2da84cf70d42c738023191248f2a4a9f7fedf6300048379bc507c763eb"} Nov 04 01:32:36.892755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.892738 4312 scope.go:122] "RemoveContainer" containerID="79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c" Nov 04 01:32:36.893087 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:36.893058 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c\": container with ID starting with 79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c not found: ID does not exist" containerID="79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c" Nov 04 01:32:36.893179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.893096 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c"} err="failed to get container status \"79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c\": rpc error: code = NotFound desc = could not find container \"79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c\": container with ID starting with 79b42fd98b51f39138afa8974ba1de3902d7a88cc863a682f63e59ddbd234d4c not found: ID does not exist" Nov 04 01:32:36.893179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.893112 4312 scope.go:122] "RemoveContainer" containerID="f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19" Nov 04 01:32:36.902484 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.902467 4312 scope.go:122] "RemoveContainer" containerID="f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19" Nov 04 01:32:36.902741 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:36.902722 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19\": container with ID starting with f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19 not found: ID does not exist" containerID="f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19" Nov 04 01:32:36.902803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.902752 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19"} err="failed to get container status \"f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19\": rpc error: code = NotFound desc = could not find container \"f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19\": container with ID starting with f701987434e51ce3ba542126e60fc365d0d49d7149dcc0ef252fa28c89954b19 not found: ID does not exist" Nov 04 01:32:36.909695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.909674 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf"] Nov 04 01:32:36.913870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.913849 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-362631-8494697844-jsmqf"] Nov 04 01:32:36.925148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.925125 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk"] Nov 04 01:32:36.929770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:36.929749 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8974fa/test-model-8974fa-kserve-6c9cfcdf7c-4fbfk"] Nov 04 01:32:37.834349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:37.834308 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" podUID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:38.110920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:38.110841 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" path="/var/lib/kubelet/pods/b5d2a781-cc39-40f3-b2fe-cd5de79d89ff/volumes" Nov 04 01:32:38.111457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:38.111440 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd92106e-d547-4052-a6a0-de9222fa375a" path="/var/lib/kubelet/pods/fd92106e-d547-4052-a6a0-de9222fa375a/volumes" Nov 04 01:32:38.522858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:38.522782 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb"] Nov 04 01:32:38.534287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:38.534248 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v"] Nov 04 01:32:39.659244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:39.659202 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" podUID="b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" containerName="payload-processing" containerID="cri-o://b4cbe68599caa72570d267460eea088bd8eef2b2523592a4017e7e974f6e3fe8" gracePeriod=25 Nov 04 01:32:39.675985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:39.675956 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" podUID="e130b444-8588-48ad-b422-54342eeae735" containerName="payload-pre-processing" containerID="cri-o://8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947" gracePeriod=25 Nov 04 01:32:39.906731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:39.906703 4312 generic.go:356] "Generic (PLEG): container finished" podID="b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" containerID="b4cbe68599caa72570d267460eea088bd8eef2b2523592a4017e7e974f6e3fe8" exitCode=0 Nov 04 01:32:39.906854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:39.906793 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" event={"ID":"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee","Type":"ContainerDied","Data":"b4cbe68599caa72570d267460eea088bd8eef2b2523592a4017e7e974f6e3fe8"} Nov 04 01:32:39.906903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:39.906882 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" event={"ID":"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee","Type":"ContainerDied","Data":"ca1e6edfec1ae484721d08b4fafe1a2cb34255ff30f7e47a2814623154cb6c60"} Nov 04 01:32:39.906903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:39.906901 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1e6edfec1ae484721d08b4fafe1a2cb34255ff30f7e47a2814623154cb6c60" Nov 04 01:32:39.940579 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:39.940513 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:32:40.085403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.085375 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-plugins-config-volume\") pod \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " Nov 04 01:32:40.085556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.085461 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-kube-api-access-zwwp4\" (UniqueName: \"kubernetes.io/projected/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-kube-api-access-zwwp4\") pod \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\" (UID: \"b5bd26f2-2287-4ae2-9640-dfbe6e1409ee\") " Nov 04 01:32:40.085839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.085814 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-plugins-config-volume" pod "b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" (UID: "b5bd26f2-2287-4ae2-9640-dfbe6e1409ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:40.087802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.087778 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-kube-api-access-zwwp4" pod "b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" (UID: "b5bd26f2-2287-4ae2-9640-dfbe6e1409ee"). InnerVolumeSpecName "kube-api-access-zwwp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:40.187090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.186951 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwwp4\" (UniqueName: \"kubernetes.io/projected/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-kube-api-access-zwwp4\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:40.187090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.186985 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:40.912315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.912283 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt" Nov 04 01:32:40.931836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.931807 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt"] Nov 04 01:32:40.935708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:40.935681 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-362631-7b65958dd-qwqqt"] Nov 04 01:32:42.109851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:42.109815 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" path="/var/lib/kubelet/pods/b5bd26f2-2287-4ae2-9640-dfbe6e1409ee/volumes" Nov 04 01:32:42.136789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:42.136755 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj"] Nov 04 01:32:42.137046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:42.137000 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" containerID="cri-o://242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1" gracePeriod=30 Nov 04 01:32:43.524001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.523943 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" podUID="f5b07a71-14ba-435f-bf23-c55c6915c384" containerName="payload-processing" containerID="cri-o://f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe" gracePeriod=25 Nov 04 01:32:43.535160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.535104 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" podUID="a7ed16d8-1905-44ae-8f12-df385a63dd88" containerName="payload-pre-processing" containerID="cri-o://541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c" gracePeriod=25 Nov 04 01:32:43.655086 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.655062 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:32:43.751368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.747865 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236a8c22-110b-4dd3-a3b0-3687494fd1dc-kube-api-access-sb8fp\" (UniqueName: \"kubernetes.io/projected/236a8c22-110b-4dd3-a3b0-3687494fd1dc-kube-api-access-sb8fp\") pod \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " Nov 04 01:32:43.751368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.747935 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/236a8c22-110b-4dd3-a3b0-3687494fd1dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236a8c22-110b-4dd3-a3b0-3687494fd1dc-plugins-config-volume\") pod \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\" (UID: \"236a8c22-110b-4dd3-a3b0-3687494fd1dc\") " Nov 04 01:32:43.751368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.749267 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236a8c22-110b-4dd3-a3b0-3687494fd1dc-plugins-config-volume" pod "236a8c22-110b-4dd3-a3b0-3687494fd1dc" (UID: "236a8c22-110b-4dd3-a3b0-3687494fd1dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:43.753289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.753257 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236a8c22-110b-4dd3-a3b0-3687494fd1dc-kube-api-access-sb8fp" pod "236a8c22-110b-4dd3-a3b0-3687494fd1dc" (UID: "236a8c22-110b-4dd3-a3b0-3687494fd1dc"). InnerVolumeSpecName "kube-api-access-sb8fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:43.762575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.762508 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb8fp\" (UniqueName: \"kubernetes.io/projected/236a8c22-110b-4dd3-a3b0-3687494fd1dc-kube-api-access-sb8fp\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:43.762575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.762539 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/236a8c22-110b-4dd3-a3b0-3687494fd1dc-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:43.790722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.790699 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:32:43.863045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.863011 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b07a71-14ba-435f-bf23-c55c6915c384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b07a71-14ba-435f-bf23-c55c6915c384-plugins-config-volume\") pod \"f5b07a71-14ba-435f-bf23-c55c6915c384\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " Nov 04 01:32:43.863188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.863125 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b07a71-14ba-435f-bf23-c55c6915c384-kube-api-access-rt66t\" (UniqueName: \"kubernetes.io/projected/f5b07a71-14ba-435f-bf23-c55c6915c384-kube-api-access-rt66t\") pod \"f5b07a71-14ba-435f-bf23-c55c6915c384\" (UID: \"f5b07a71-14ba-435f-bf23-c55c6915c384\") " Nov 04 01:32:43.863387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.863366 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b07a71-14ba-435f-bf23-c55c6915c384-plugins-config-volume" pod "f5b07a71-14ba-435f-bf23-c55c6915c384" (UID: "f5b07a71-14ba-435f-bf23-c55c6915c384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:43.863553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.863532 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b07a71-14ba-435f-bf23-c55c6915c384-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:43.865106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.865085 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b07a71-14ba-435f-bf23-c55c6915c384-kube-api-access-rt66t" pod "f5b07a71-14ba-435f-bf23-c55c6915c384" (UID: "f5b07a71-14ba-435f-bf23-c55c6915c384"). InnerVolumeSpecName "kube-api-access-rt66t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:43.933246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.933161 4312 generic.go:356] "Generic (PLEG): container finished" podID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" containerID="fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0" exitCode=0 Nov 04 01:32:43.933246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.933229 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" event={"ID":"236a8c22-110b-4dd3-a3b0-3687494fd1dc","Type":"ContainerDied","Data":"fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0"} Nov 04 01:32:43.933492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.933236 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" Nov 04 01:32:43.933492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.933327 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq" event={"ID":"236a8c22-110b-4dd3-a3b0-3687494fd1dc","Type":"ContainerDied","Data":"7d1289cc3fa5f607385cd0c86ce21bc01609906b2c3401f53bf87f5d4a2f7bcd"} Nov 04 01:32:43.933492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.933393 4312 scope.go:122] "RemoveContainer" containerID="fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0" Nov 04 01:32:43.934706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.934676 4312 generic.go:356] "Generic (PLEG): container finished" podID="f5b07a71-14ba-435f-bf23-c55c6915c384" containerID="f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe" exitCode=0 Nov 04 01:32:43.934809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.934702 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" event={"ID":"f5b07a71-14ba-435f-bf23-c55c6915c384","Type":"ContainerDied","Data":"f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe"} Nov 04 01:32:43.934809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.934741 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" Nov 04 01:32:43.934809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.934749 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb" event={"ID":"f5b07a71-14ba-435f-bf23-c55c6915c384","Type":"ContainerDied","Data":"dbded85416f31e27af47b9e55ef4ec807bf6cf77b8cebd0d13f2fd11865ba949"} Nov 04 01:32:43.946049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.946033 4312 scope.go:122] "RemoveContainer" containerID="fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0" Nov 04 01:32:43.946387 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:43.946355 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0\": container with ID starting with fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0 not found: ID does not exist" containerID="fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0" Nov 04 01:32:43.946504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.946394 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0"} err="failed to get container status \"fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0\": rpc error: code = NotFound desc = could not find container \"fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0\": container with ID starting with fd8d91a011a690b580667cca49acd2fad0eeac6c76f205bcef289c7e515212d0 not found: ID does not exist" Nov 04 01:32:43.946504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.946412 4312 scope.go:122] "RemoveContainer" containerID="f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe" Nov 04 01:32:43.956093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.956074 4312 scope.go:122] "RemoveContainer" containerID="f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe" Nov 04 01:32:43.956416 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:43.956398 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe\": container with ID starting with f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe not found: ID does not exist" containerID="f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe" Nov 04 01:32:43.956476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.956423 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe"} err="failed to get container status \"f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe\": rpc error: code = NotFound desc = could not find container \"f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe\": container with ID starting with f638886158a2be257779a713563d9569cf21e3d5e8bea076e072a62583aab7fe not found: ID does not exist" Nov 04 01:32:43.958903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.958874 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq"] Nov 04 01:32:43.964978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.964954 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt66t\" (UniqueName: \"kubernetes.io/projected/f5b07a71-14ba-435f-bf23-c55c6915c384-kube-api-access-rt66t\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:43.965496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.965476 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a362b6-6c76f5f58c-c2pkq"] Nov 04 01:32:43.982171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.982152 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb"] Nov 04 01:32:43.987916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:43.987893 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8974fa-74b6b89554-n5mjb"] Nov 04 01:32:44.109245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:44.109217 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" path="/var/lib/kubelet/pods/236a8c22-110b-4dd3-a3b0-3687494fd1dc/volumes" Nov 04 01:32:44.109671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:44.109657 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b07a71-14ba-435f-bf23-c55c6915c384" path="/var/lib/kubelet/pods/f5b07a71-14ba-435f-bf23-c55c6915c384/volumes" Nov 04 01:32:44.722246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:44.722174 4312 patch_prober.go:29] interesting pod/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:32:44.722246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:44.722229 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:32:45.028077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.028039 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" podUID="a7ed16d8-1905-44ae-8f12-df385a63dd88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:45.066127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.066093 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf"] Nov 04 01:32:45.066415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.066381 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" podUID="757cedc8-09f9-4db9-9a99-72b42341e1f1" containerName="maas-api" containerID="cri-o://e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec" gracePeriod=30 Nov 04 01:32:45.366185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.366157 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:32:45.409420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.409390 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-maas-api-tls\") pod \"757cedc8-09f9-4db9-9a99-72b42341e1f1\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " Nov 04 01:32:45.409596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.409445 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-metrics-certs\") pod \"757cedc8-09f9-4db9-9a99-72b42341e1f1\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " Nov 04 01:32:45.409596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.409557 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757cedc8-09f9-4db9-9a99-72b42341e1f1-kube-api-access-sfpn5\" (UniqueName: \"kubernetes.io/projected/757cedc8-09f9-4db9-9a99-72b42341e1f1-kube-api-access-sfpn5\") pod \"757cedc8-09f9-4db9-9a99-72b42341e1f1\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " Nov 04 01:32:45.409817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.409649 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757cedc8-09f9-4db9-9a99-72b42341e1f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757cedc8-09f9-4db9-9a99-72b42341e1f1-openshift-service-ca\") pod \"757cedc8-09f9-4db9-9a99-72b42341e1f1\" (UID: \"757cedc8-09f9-4db9-9a99-72b42341e1f1\") " Nov 04 01:32:45.409994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.409959 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757cedc8-09f9-4db9-9a99-72b42341e1f1-openshift-service-ca" pod "757cedc8-09f9-4db9-9a99-72b42341e1f1" (UID: "757cedc8-09f9-4db9-9a99-72b42341e1f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:45.410150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.410132 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757cedc8-09f9-4db9-9a99-72b42341e1f1-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.411890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.411781 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757cedc8-09f9-4db9-9a99-72b42341e1f1-kube-api-access-sfpn5" pod "757cedc8-09f9-4db9-9a99-72b42341e1f1" (UID: "757cedc8-09f9-4db9-9a99-72b42341e1f1"). InnerVolumeSpecName "kube-api-access-sfpn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:45.412762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.412728 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-metrics-certs" pod "757cedc8-09f9-4db9-9a99-72b42341e1f1" (UID: "757cedc8-09f9-4db9-9a99-72b42341e1f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:45.412940 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.412918 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-maas-api-tls" pod "757cedc8-09f9-4db9-9a99-72b42341e1f1" (UID: "757cedc8-09f9-4db9-9a99-72b42341e1f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:45.534356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.511730 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpn5\" (UniqueName: \"kubernetes.io/projected/757cedc8-09f9-4db9-9a99-72b42341e1f1-kube-api-access-sfpn5\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.534356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.530360 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.534356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.530419 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757cedc8-09f9-4db9-9a99-72b42341e1f1-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.952406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.952373 4312 generic.go:356] "Generic (PLEG): container finished" podID="757cedc8-09f9-4db9-9a99-72b42341e1f1" containerID="e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec" exitCode=0 Nov 04 01:32:45.952838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.952422 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" Nov 04 01:32:45.952838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.952465 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" event={"ID":"757cedc8-09f9-4db9-9a99-72b42341e1f1","Type":"ContainerDied","Data":"e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec"} Nov 04 01:32:45.952838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.952521 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf" event={"ID":"757cedc8-09f9-4db9-9a99-72b42341e1f1","Type":"ContainerDied","Data":"abd8116d3420232713202b198ae208502a123622529f6e6b0750a3cc0c5f8087"} Nov 04 01:32:45.952838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.952544 4312 scope.go:122] "RemoveContainer" containerID="e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec" Nov 04 01:32:45.963257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.963240 4312 scope.go:122] "RemoveContainer" containerID="e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec" Nov 04 01:32:45.963590 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:45.963563 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec\": container with ID starting with e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec not found: ID does not exist" containerID="e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec" Nov 04 01:32:45.963688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.963598 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec"} err="failed to get container status \"e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec\": rpc error: code = NotFound desc = could not find container \"e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec\": container with ID starting with e9a86c14c48591f0697620eb374d0c308deca7fecbee557c1880220f2eac34ec not found: ID does not exist" Nov 04 01:32:45.999695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:45.999674 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf"] Nov 04 01:32:46.041455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:46.041433 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8974fa-64fc54c5-nv4jf"] Nov 04 01:32:46.109817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:46.109789 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757cedc8-09f9-4db9-9a99-72b42341e1f1" path="/var/lib/kubelet/pods/757cedc8-09f9-4db9-9a99-72b42341e1f1/volumes" Nov 04 01:32:47.204877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.204847 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" podUID="e130b444-8588-48ad-b422-54342eeae735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:47.394712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.394686 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:32:47.551701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.551672 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-certs\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.551888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.551749 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istiod-ca-cert\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.551888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.551808 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-credential-socket\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.551888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.551844 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-kube-api-access-wqcmc\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-kube-api-access-wqcmc\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.551888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.551867 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-envoy\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.552124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.551935 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-token\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-token\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.552124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.551987 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-podinfo\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.552124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552011 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-certs" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.552124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552067 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-data\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.552475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552134 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-credential-socket" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.552475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552056 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istiod-ca-cert" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:47.552475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552180 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-socket\") pod \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\" (UID: \"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4\") " Nov 04 01:32:47.552475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552369 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-data" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.552475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552430 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-socket" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.552916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552891 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.552998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552921 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.552998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552937 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.552998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552951 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.552998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.552968 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.554107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.554078 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-envoy" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.554209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.554143 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-token" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:47.554209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.554156 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-podinfo" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:47.554292 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.554205 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-kube-api-access-wqcmc" pod "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" (UID: "e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4"). InnerVolumeSpecName "kube-api-access-wqcmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:47.653615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.653581 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqcmc\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-kube-api-access-wqcmc\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.653615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.653611 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.653615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.653621 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.653857 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.653629 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.968393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.968286 4312 generic.go:356] "Generic (PLEG): container finished" podID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerID="242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1" exitCode=0 Nov 04 01:32:47.968393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.968378 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" Nov 04 01:32:47.968623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.968370 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" event={"ID":"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4","Type":"ContainerDied","Data":"242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1"} Nov 04 01:32:47.968623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.968536 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj" event={"ID":"e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4","Type":"ContainerDied","Data":"cb314f82729636a2fa05a2b1dc0f66b1476a42fa36d030d3a8f6048f53aedb27"} Nov 04 01:32:47.968623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.968573 4312 scope.go:122] "RemoveContainer" containerID="242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1" Nov 04 01:32:47.979537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.979518 4312 scope.go:122] "RemoveContainer" containerID="771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280" Nov 04 01:32:47.988977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.988960 4312 scope.go:122] "RemoveContainer" containerID="242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1" Nov 04 01:32:47.989293 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:47.989270 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1\": container with ID starting with 242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1 not found: ID does not exist" containerID="242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1" Nov 04 01:32:47.989384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.989299 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1"} err="failed to get container status \"242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1\": rpc error: code = NotFound desc = could not find container \"242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1\": container with ID starting with 242448e1e5556eb949eafe270fc24dc89ada4e3e9c4f815df446c75e01d255d1 not found: ID does not exist" Nov 04 01:32:47.989384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.989316 4312 scope.go:122] "RemoveContainer" containerID="771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280" Nov 04 01:32:47.989578 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:47.989562 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280\": container with ID starting with 771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280 not found: ID does not exist" containerID="771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280" Nov 04 01:32:47.989626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.989582 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280"} err="failed to get container status \"771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280\": rpc error: code = NotFound desc = could not find container \"771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280\": container with ID starting with 771ac8f33a41009b64af60888605bab506d13da7e6a21e7e5d811e0a27748280 not found: ID does not exist" Nov 04 01:32:47.995866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:47.995846 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj"] Nov 04 01:32:48.001318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:48.001288 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-362631-openshift-default-5c886fdbcf-b24jj"] Nov 04 01:32:48.109524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:48.109494 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" path="/var/lib/kubelet/pods/e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4/volumes" Nov 04 01:32:49.579813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:49.579776 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj"] Nov 04 01:32:49.597895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:49.597862 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx"] Nov 04 01:32:53.038360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.033413 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7"] Nov 04 01:32:53.038360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.034042 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" podUID="9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" containerName="maas-api" containerID="cri-o://44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1" gracePeriod=30 Nov 04 01:32:53.279021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.278992 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw"] Nov 04 01:32:53.279244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.279220 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" containerID="cri-o://cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26" gracePeriod=30 Nov 04 01:32:53.294106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.294048 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs"] Nov 04 01:32:53.294317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.294293 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" containerID="cri-o://cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc" gracePeriod=30 Nov 04 01:32:53.306440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.306415 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:32:53.433200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.433171 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-metrics-certs\") pod \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " Nov 04 01:32:53.433390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.433285 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-openshift-service-ca\") pod \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " Nov 04 01:32:53.433390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.433321 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-kube-api-access-vwz7n\" (UniqueName: \"kubernetes.io/projected/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-kube-api-access-vwz7n\") pod \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " Nov 04 01:32:53.433390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.433373 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-maas-api-tls\") pod \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\" (UID: \"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c\") " Nov 04 01:32:53.433695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.433668 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-openshift-service-ca" pod "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" (UID: "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:53.433904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.433883 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.435380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.435356 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-metrics-certs" pod "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" (UID: "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:53.435497 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.435477 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-maas-api-tls" pod "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" (UID: "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:53.435553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.435523 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-kube-api-access-vwz7n" pod "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" (UID: "9beada2e-b29f-4762-8e9d-fd99c4a4fe0c"). InnerVolumeSpecName "kube-api-access-vwz7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:53.535155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.535123 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwz7n\" (UniqueName: \"kubernetes.io/projected/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-kube-api-access-vwz7n\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.535155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.535149 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.535155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:53.535160 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.014559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.014526 4312 generic.go:356] "Generic (PLEG): container finished" podID="9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" containerID="44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1" exitCode=0 Nov 04 01:32:54.014734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.014586 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" Nov 04 01:32:54.014734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.014611 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" event={"ID":"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c","Type":"ContainerDied","Data":"44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1"} Nov 04 01:32:54.014734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.014707 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7" event={"ID":"9beada2e-b29f-4762-8e9d-fd99c4a4fe0c","Type":"ContainerDied","Data":"d7359a3caa320319bc5d2386b9c5bd0f85eca2a6648d1b2ef33f61f2b8d4aefc"} Nov 04 01:32:54.014915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.014736 4312 scope.go:122] "RemoveContainer" containerID="44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1" Nov 04 01:32:54.025076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.025059 4312 scope.go:122] "RemoveContainer" containerID="44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1" Nov 04 01:32:54.025344 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:54.025314 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1\": container with ID starting with 44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1 not found: ID does not exist" containerID="44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1" Nov 04 01:32:54.025402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.025351 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1"} err="failed to get container status \"44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1\": rpc error: code = NotFound desc = could not find container \"44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1\": container with ID starting with 44fb0911c7907c9cacc5d33e6324e14d46bfa1d3805e735427b5d9da9acdb0a1 not found: ID does not exist" Nov 04 01:32:54.037679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.037659 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7"] Nov 04 01:32:54.042431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.042410 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-23ef8f-869547d9dd-2cpj7"] Nov 04 01:32:54.107566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.107534 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" path="/var/lib/kubelet/pods/9beada2e-b29f-4762-8e9d-fd99c4a4fe0c/volumes" Nov 04 01:32:54.580868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.580810 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" podUID="ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" containerName="payload-processing" containerID="cri-o://ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8" gracePeriod=25 Nov 04 01:32:54.598313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.598275 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" podUID="20df1fd2-d5ca-4a9a-8280-fed7277e18ac" containerName="payload-pre-processing" containerID="cri-o://8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915" gracePeriod=25 Nov 04 01:32:54.857810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.857783 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:32:54.891093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.891064 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:32:54.951161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951135 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-plugins-config-volume\") pod \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " Nov 04 01:32:54.951320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951206 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-plugins-config-volume\") pod \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " Nov 04 01:32:54.951320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951258 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-kube-api-access-7qccc\" (UniqueName: \"kubernetes.io/projected/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-kube-api-access-7qccc\") pod \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\" (UID: \"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6\") " Nov 04 01:32:54.951320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951284 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-kube-api-access-xhrgp\" (UniqueName: \"kubernetes.io/projected/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-kube-api-access-xhrgp\") pod \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\" (UID: \"20df1fd2-d5ca-4a9a-8280-fed7277e18ac\") " Nov 04 01:32:54.951588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951560 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-plugins-config-volume" pod "20df1fd2-d5ca-4a9a-8280-fed7277e18ac" (UID: "20df1fd2-d5ca-4a9a-8280-fed7277e18ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:54.951646 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951626 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-plugins-config-volume" pod "ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" (UID: "ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:54.951736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951721 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.951786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.951739 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.953399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.953381 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-kube-api-access-xhrgp" pod "20df1fd2-d5ca-4a9a-8280-fed7277e18ac" (UID: "20df1fd2-d5ca-4a9a-8280-fed7277e18ac"). InnerVolumeSpecName "kube-api-access-xhrgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:54.953463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:54.953410 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-kube-api-access-7qccc" pod "ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" (UID: "ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6"). InnerVolumeSpecName "kube-api-access-7qccc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:55.024192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.024159 4312 generic.go:356] "Generic (PLEG): container finished" podID="ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" containerID="ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8" exitCode=0 Nov 04 01:32:55.024394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.024203 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" Nov 04 01:32:55.024394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.024242 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" event={"ID":"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6","Type":"ContainerDied","Data":"ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8"} Nov 04 01:32:55.024394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.024301 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj" event={"ID":"ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6","Type":"ContainerDied","Data":"cf2ff512ef75b9e76016ed87683e29d359a69bbdb329cbc4cfceb9cf576bf679"} Nov 04 01:32:55.024394 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.024357 4312 scope.go:122] "RemoveContainer" containerID="ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8" Nov 04 01:32:55.036214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.036183 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" podUID="a7ed16d8-1905-44ae-8f12-df385a63dd88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:55.039711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.039681 4312 generic.go:356] "Generic (PLEG): container finished" podID="20df1fd2-d5ca-4a9a-8280-fed7277e18ac" containerID="8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915" exitCode=0 Nov 04 01:32:55.039978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.039908 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" event={"ID":"20df1fd2-d5ca-4a9a-8280-fed7277e18ac","Type":"ContainerDied","Data":"8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915"} Nov 04 01:32:55.039978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.039961 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" event={"ID":"20df1fd2-d5ca-4a9a-8280-fed7277e18ac","Type":"ContainerDied","Data":"30d9bbf3d82b08d6d3ce924fa2c9636c67de7866d80d721c15c55c97b77b2993"} Nov 04 01:32:55.040123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.040036 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx" Nov 04 01:32:55.049096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.049073 4312 scope.go:122] "RemoveContainer" containerID="ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8" Nov 04 01:32:55.049521 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:55.049489 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8\": container with ID starting with ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8 not found: ID does not exist" containerID="ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8" Nov 04 01:32:55.049599 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.049516 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8"} err="failed to get container status \"ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8\": rpc error: code = NotFound desc = could not find container \"ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8\": container with ID starting with ba4161b390ac186906499b9d35dafd5369988996813a20775fa006aa62b335e8 not found: ID does not exist" Nov 04 01:32:55.049599 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.049534 4312 scope.go:122] "RemoveContainer" containerID="8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915" Nov 04 01:32:55.052650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.052629 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qccc\" (UniqueName: \"kubernetes.io/projected/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6-kube-api-access-7qccc\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:55.052742 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.052652 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhrgp\" (UniqueName: \"kubernetes.io/projected/20df1fd2-d5ca-4a9a-8280-fed7277e18ac-kube-api-access-xhrgp\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:55.058446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.058424 4312 scope.go:122] "RemoveContainer" containerID="8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915" Nov 04 01:32:55.059065 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:55.058889 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915\": container with ID starting with 8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915 not found: ID does not exist" containerID="8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915" Nov 04 01:32:55.059065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.058914 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915"} err="failed to get container status \"8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915\": rpc error: code = NotFound desc = could not find container \"8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915\": container with ID starting with 8de53a58be73c120ad1b1ba74480314cadb8993e644feadf561b490942a65915 not found: ID does not exist" Nov 04 01:32:55.061535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.061514 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj"] Nov 04 01:32:55.065347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.065314 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-23ef8f-7f44574cb-fcqhj"] Nov 04 01:32:55.076555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.076530 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx"] Nov 04 01:32:55.081913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.081894 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-23ef8f-689978d947-pd6qx"] Nov 04 01:32:55.518017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.517982 4312 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:32:55.518184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:55.518033 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:32:56.107647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:56.107613 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20df1fd2-d5ca-4a9a-8280-fed7277e18ac" path="/var/lib/kubelet/pods/20df1fd2-d5ca-4a9a-8280-fed7277e18ac/volumes" Nov 04 01:32:56.108168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:56.108150 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" path="/var/lib/kubelet/pods/ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6/volumes" Nov 04 01:32:57.185501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:57.185467 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf"] Nov 04 01:32:57.185982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:57.185696 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" containerID="cri-o://cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5" gracePeriod=30 Nov 04 01:32:57.205667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:57.205633 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" podUID="e130b444-8588-48ad-b422-54342eeae735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:58.566256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.566237 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:32:58.583198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.583171 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:32:58.595767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.595742 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-kube-api-access-svrbm\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-kube-api-access-svrbm\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.595929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.595851 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-token\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-token\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.595929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.595888 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-certs\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.596054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.595969 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d807e2eb-432b-4219-814d-d5bbd21a4c53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d807e2eb-432b-4219-814d-d5bbd21a4c53-istiod-ca-cert\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.596054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.596004 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-data\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.596150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.596057 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-credential-socket\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.596349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.596306 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-certs" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.596496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.596363 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-credential-socket" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.596496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.596416 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-socket\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.603355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.596675 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-data" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.603355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.598113 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-socket" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.603355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.598395 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d807e2eb-432b-4219-814d-d5bbd21a4c53-istiod-ca-cert" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:58.603355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.598697 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-podinfo\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.603355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.598847 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-envoy\") pod \"d807e2eb-432b-4219-814d-d5bbd21a4c53\" (UID: \"d807e2eb-432b-4219-814d-d5bbd21a4c53\") " Nov 04 01:32:58.603355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.598879 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-kube-api-access-svrbm" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "kube-api-access-svrbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:58.603355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.601030 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-podinfo" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:58.603807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.603681 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-envoy" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.605237 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605209 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d807e2eb-432b-4219-814d-d5bbd21a4c53-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.605354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605240 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.605354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605255 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.605354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605268 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.605354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605285 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.605354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605298 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.605354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605312 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrbm\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-kube-api-access-svrbm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.605354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.605326 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d807e2eb-432b-4219-814d-d5bbd21a4c53-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.612903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.612807 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-token" pod "d807e2eb-432b-4219-814d-d5bbd21a4c53" (UID: "d807e2eb-432b-4219-814d-d5bbd21a4c53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:58.706206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706171 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-token\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-token\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706265 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-credential-socket\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706293 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-socket\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706369 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-certs\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706429 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-envoy\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706479 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1cbc30-9a61-4274-b2a4-9da61d518423-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1cbc30-9a61-4274-b2a4-9da61d518423-istiod-ca-cert\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706522 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-kube-api-access-g7pbw\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-kube-api-access-g7pbw\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706581 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-data\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706575 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-credential-socket" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.706680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706651 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-podinfo\") pod \"be1cbc30-9a61-4274-b2a4-9da61d518423\" (UID: \"be1cbc30-9a61-4274-b2a4-9da61d518423\") " Nov 04 01:32:58.706680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706670 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-socket" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.706896 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706684 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-certs" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.706962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.706896 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-data" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.707038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.707006 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1cbc30-9a61-4274-b2a4-9da61d518423-istiod-ca-cert" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:58.707185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.707163 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1cbc30-9a61-4274-b2a4-9da61d518423-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.707185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.707181 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.707288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.707197 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d807e2eb-432b-4219-814d-d5bbd21a4c53-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.707288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.707207 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.707288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.707220 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.707288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.707232 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.709058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.709037 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-podinfo" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:58.709058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.709042 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-envoy" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:58.709214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.709076 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-token" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:58.709214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.709086 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-kube-api-access-g7pbw" pod "be1cbc30-9a61-4274-b2a4-9da61d518423" (UID: "be1cbc30-9a61-4274-b2a4-9da61d518423"). InnerVolumeSpecName "kube-api-access-g7pbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:58.808898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.808804 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.808898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.808839 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.808898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.808849 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1cbc30-9a61-4274-b2a4-9da61d518423-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:58.808898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:58.808858 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7pbw\" (UniqueName: \"kubernetes.io/projected/be1cbc30-9a61-4274-b2a4-9da61d518423-kube-api-access-g7pbw\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:59.067908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.067820 4312 generic.go:356] "Generic (PLEG): container finished" podID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerID="cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26" exitCode=0 Nov 04 01:32:59.067908 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.067888 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" Nov 04 01:32:59.068117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.067904 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" event={"ID":"d807e2eb-432b-4219-814d-d5bbd21a4c53","Type":"ContainerDied","Data":"cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26"} Nov 04 01:32:59.068117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.067986 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw" event={"ID":"d807e2eb-432b-4219-814d-d5bbd21a4c53","Type":"ContainerDied","Data":"499b0dd0f483285c12ff6f426b1bcc4927b4a55c0dfa3fe47b050848634972d8"} Nov 04 01:32:59.068117 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.068012 4312 scope.go:122] "RemoveContainer" containerID="cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26" Nov 04 01:32:59.069520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.069483 4312 generic.go:356] "Generic (PLEG): container finished" podID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerID="cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc" exitCode=0 Nov 04 01:32:59.069648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.069549 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" Nov 04 01:32:59.069648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.069554 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" event={"ID":"be1cbc30-9a61-4274-b2a4-9da61d518423","Type":"ContainerDied","Data":"cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc"} Nov 04 01:32:59.069648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.069605 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs" event={"ID":"be1cbc30-9a61-4274-b2a4-9da61d518423","Type":"ContainerDied","Data":"8ab2ba5f2fca1bd9bc725f85e37ee13c05e9d5fb1d98047d3e41264b307ca7b1"} Nov 04 01:32:59.080592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.080573 4312 scope.go:122] "RemoveContainer" containerID="6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b" Nov 04 01:32:59.089695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.089677 4312 scope.go:122] "RemoveContainer" containerID="cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26" Nov 04 01:32:59.090008 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:59.089990 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26\": container with ID starting with cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26 not found: ID does not exist" containerID="cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26" Nov 04 01:32:59.090070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.090017 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26"} err="failed to get container status \"cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26\": rpc error: code = NotFound desc = could not find container \"cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26\": container with ID starting with cb9f3b10c8f957404d4870b88065d98e709e6aa55d2cd043b65b3ab8f3196a26 not found: ID does not exist" Nov 04 01:32:59.090070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.090033 4312 scope.go:122] "RemoveContainer" containerID="6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b" Nov 04 01:32:59.090284 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:59.090262 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b\": container with ID starting with 6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b not found: ID does not exist" containerID="6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b" Nov 04 01:32:59.090356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.090294 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b"} err="failed to get container status \"6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b\": rpc error: code = NotFound desc = could not find container \"6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b\": container with ID starting with 6c43fa36e795067d787d2e9e53948c62ce38377c1ca3427adb2926831ed3323b not found: ID does not exist" Nov 04 01:32:59.090356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.090318 4312 scope.go:122] "RemoveContainer" containerID="cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc" Nov 04 01:32:59.095711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.095683 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw"] Nov 04 01:32:59.101019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.100990 4312 scope.go:122] "RemoveContainer" containerID="cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc" Nov 04 01:32:59.101454 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:59.101420 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc\": container with ID starting with cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc not found: ID does not exist" containerID="cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc" Nov 04 01:32:59.101566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.101463 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc"} err="failed to get container status \"cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc\": rpc error: code = NotFound desc = could not find container \"cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc\": container with ID starting with cd3adfa33afc9a3eadba5d97553bb02a502e840ba568782335c7bfa545c430cc not found: ID does not exist" Nov 04 01:32:59.103594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.103567 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-8vbbw"] Nov 04 01:32:59.114663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.114629 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs"] Nov 04 01:32:59.120565 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.120542 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-njjzs"] Nov 04 01:32:59.164574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.164539 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj"] Nov 04 01:32:59.213074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.213030 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7"] Nov 04 01:32:59.458570 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:32:59.458475 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:32:59.940818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:32:59.940788 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:33:00.022485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.022454 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e130b444-8588-48ad-b422-54342eeae735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e130b444-8588-48ad-b422-54342eeae735-plugins-config-volume\") pod \"e130b444-8588-48ad-b422-54342eeae735\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " Nov 04 01:33:00.022669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.022652 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e130b444-8588-48ad-b422-54342eeae735-kube-api-access-msv57\" (UniqueName: \"kubernetes.io/projected/e130b444-8588-48ad-b422-54342eeae735-kube-api-access-msv57\") pod \"e130b444-8588-48ad-b422-54342eeae735\" (UID: \"e130b444-8588-48ad-b422-54342eeae735\") " Nov 04 01:33:00.022858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.022835 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e130b444-8588-48ad-b422-54342eeae735-plugins-config-volume" pod "e130b444-8588-48ad-b422-54342eeae735" (UID: "e130b444-8588-48ad-b422-54342eeae735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:00.022961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.022948 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e130b444-8588-48ad-b422-54342eeae735-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:00.024674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.024654 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e130b444-8588-48ad-b422-54342eeae735-kube-api-access-msv57" pod "e130b444-8588-48ad-b422-54342eeae735" (UID: "e130b444-8588-48ad-b422-54342eeae735"). InnerVolumeSpecName "kube-api-access-msv57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:00.076981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.076893 4312 generic.go:356] "Generic (PLEG): container finished" podID="e130b444-8588-48ad-b422-54342eeae735" containerID="8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947" exitCode=0 Nov 04 01:33:00.076981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.076939 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" Nov 04 01:33:00.077214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.076974 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" event={"ID":"e130b444-8588-48ad-b422-54342eeae735","Type":"ContainerDied","Data":"8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947"} Nov 04 01:33:00.077214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.077030 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb" event={"ID":"e130b444-8588-48ad-b422-54342eeae735","Type":"ContainerDied","Data":"dfc16052aafa3268d461af19d0b7eff7cdbc24f9ec929e81536c6d0eb839c0e0"} Nov 04 01:33:00.077214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.077058 4312 scope.go:122] "RemoveContainer" containerID="8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947" Nov 04 01:33:00.087713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.087691 4312 scope.go:122] "RemoveContainer" containerID="8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947" Nov 04 01:33:00.088017 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:00.087995 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947\": container with ID starting with 8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947 not found: ID does not exist" containerID="8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947" Nov 04 01:33:00.088101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.088022 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947"} err="failed to get container status \"8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947\": rpc error: code = NotFound desc = could not find container \"8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947\": container with ID starting with 8dd1b4cdbf37a3e14a6b1d966f4dfb1732f315c1b921b9e9ac6ebca61ea8e947 not found: ID does not exist" Nov 04 01:33:00.108256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.108230 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" path="/var/lib/kubelet/pods/be1cbc30-9a61-4274-b2a4-9da61d518423/volumes" Nov 04 01:33:00.108764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.108748 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" path="/var/lib/kubelet/pods/d807e2eb-432b-4219-814d-d5bbd21a4c53/volumes" Nov 04 01:33:00.109044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.109031 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb"] Nov 04 01:33:00.109093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.109060 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-362631-5b88d89f5f-qdbcb"] Nov 04 01:33:00.124452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:00.124427 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msv57\" (UniqueName: \"kubernetes.io/projected/e130b444-8588-48ad-b422-54342eeae735-kube-api-access-msv57\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.110486 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.110449 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e130b444-8588-48ad-b422-54342eeae735" path="/var/lib/kubelet/pods/e130b444-8588-48ad-b422-54342eeae735/volumes" Nov 04 01:33:02.120327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.120295 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:02.120494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.120377 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:02.452765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.452736 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:33:02.551511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551475 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-envoy\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551514 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-kube-api-access-dmq6d\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-kube-api-access-dmq6d\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551534 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-token\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-token\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551581 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-data\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551613 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istiod-ca-cert\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551774 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551718 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-podinfo\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551779 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-credential-socket\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551856 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-socket\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551905 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-certs\") pod \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\" (UID: \"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa\") " Nov 04 01:33:02.551992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.551966 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-data" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:02.552209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552075 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istiod-ca-cert" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:02.552275 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552204 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-credential-socket" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:02.552376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552303 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-certs" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:02.552467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552447 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-socket" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:02.552620 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552604 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.552683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552627 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.552683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552643 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.552683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552658 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.552683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.552671 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.553939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.553912 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-kube-api-access-dmq6d" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "kube-api-access-dmq6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:02.554053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.553978 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-token" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:02.554185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.554168 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-podinfo" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:02.554185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.554175 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-envoy" pod "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" (UID: "a1d79be2-e8ca-417a-bd4b-aa14b8a237fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:02.653693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.653661 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.653693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.653689 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.653693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.653698 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmq6d\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-kube-api-access-dmq6d\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.653921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.653708 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.712491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.712409 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk"] Nov 04 01:33:02.712732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:02.712684 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" podUID="44ea9217-3aeb-4591-a754-425c549c29f8" containerName="maas-api" containerID="cri-o://eb8664ddf8ce83c894011765e0df3b5620d966cfd31ff4e11711189345984b2b" gracePeriod=30 Nov 04 01:33:03.116033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.112357 4312 generic.go:356] "Generic (PLEG): container finished" podID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerID="cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5" exitCode=0 Nov 04 01:33:03.116033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.112473 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" Nov 04 01:33:03.116033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.112596 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" event={"ID":"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa","Type":"ContainerDied","Data":"cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5"} Nov 04 01:33:03.116033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.112651 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf" event={"ID":"a1d79be2-e8ca-417a-bd4b-aa14b8a237fa","Type":"ContainerDied","Data":"5e18d172a91fdbb9a74928d7f14d29b54cfa6c064a9d174ad063753ac4421e89"} Nov 04 01:33:03.116033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.112682 4312 scope.go:122] "RemoveContainer" containerID="cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5" Nov 04 01:33:03.118761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.118714 4312 generic.go:356] "Generic (PLEG): container finished" podID="44ea9217-3aeb-4591-a754-425c549c29f8" containerID="eb8664ddf8ce83c894011765e0df3b5620d966cfd31ff4e11711189345984b2b" exitCode=0 Nov 04 01:33:03.118986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.118810 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" event={"ID":"44ea9217-3aeb-4591-a754-425c549c29f8","Type":"ContainerDied","Data":"eb8664ddf8ce83c894011765e0df3b5620d966cfd31ff4e11711189345984b2b"} Nov 04 01:33:03.127353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.127322 4312 scope.go:122] "RemoveContainer" containerID="10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a" Nov 04 01:33:03.136583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.136552 4312 scope.go:122] "RemoveContainer" containerID="cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5" Nov 04 01:33:03.136908 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:03.136887 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5\": container with ID starting with cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5 not found: ID does not exist" containerID="cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5" Nov 04 01:33:03.136981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.136915 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5"} err="failed to get container status \"cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5\": rpc error: code = NotFound desc = could not find container \"cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5\": container with ID starting with cde08478aae3e44f7ede812a6cfa878b3756cfbdebbd31203133dff5153d04e5 not found: ID does not exist" Nov 04 01:33:03.136981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.136934 4312 scope.go:122] "RemoveContainer" containerID="10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a" Nov 04 01:33:03.137181 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:03.137158 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a\": container with ID starting with 10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a not found: ID does not exist" containerID="10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a" Nov 04 01:33:03.137256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.137191 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a"} err="failed to get container status \"10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a\": rpc error: code = NotFound desc = could not find container \"10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a\": container with ID starting with 10b33911dff99467f34c667726181f93e80a59cb34a9351f2159d45d59bbe71a not found: ID does not exist" Nov 04 01:33:03.144457 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.144430 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf"] Nov 04 01:33:03.149323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.149303 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-23ef8f-openshift-default-59d9b7799f-dwvkf"] Nov 04 01:33:03.163695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.163673 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:33:03.260004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.259910 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls\") pod \"44ea9217-3aeb-4591-a754-425c549c29f8\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " Nov 04 01:33:03.260004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.259988 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ea9217-3aeb-4591-a754-425c549c29f8-kube-api-access-6pzxl\" (UniqueName: \"kubernetes.io/projected/44ea9217-3aeb-4591-a754-425c549c29f8-kube-api-access-6pzxl\") pod \"44ea9217-3aeb-4591-a754-425c549c29f8\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " Nov 04 01:33:03.260196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.260055 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ea9217-3aeb-4591-a754-425c549c29f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ea9217-3aeb-4591-a754-425c549c29f8-openshift-service-ca\") pod \"44ea9217-3aeb-4591-a754-425c549c29f8\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " Nov 04 01:33:03.260196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.260123 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-metrics-certs\") pod \"44ea9217-3aeb-4591-a754-425c549c29f8\" (UID: \"44ea9217-3aeb-4591-a754-425c549c29f8\") " Nov 04 01:33:03.260503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.260476 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ea9217-3aeb-4591-a754-425c549c29f8-openshift-service-ca" pod "44ea9217-3aeb-4591-a754-425c549c29f8" (UID: "44ea9217-3aeb-4591-a754-425c549c29f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:03.262109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.262085 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-metrics-certs" pod "44ea9217-3aeb-4591-a754-425c549c29f8" (UID: "44ea9217-3aeb-4591-a754-425c549c29f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:03.262209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.262114 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls" pod "44ea9217-3aeb-4591-a754-425c549c29f8" (UID: "44ea9217-3aeb-4591-a754-425c549c29f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:03.262209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.262175 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ea9217-3aeb-4591-a754-425c549c29f8-kube-api-access-6pzxl" pod "44ea9217-3aeb-4591-a754-425c549c29f8" (UID: "44ea9217-3aeb-4591-a754-425c549c29f8"). InnerVolumeSpecName "kube-api-access-6pzxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:03.361204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.361167 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:03.361204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.361205 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzxl\" (UniqueName: \"kubernetes.io/projected/44ea9217-3aeb-4591-a754-425c549c29f8-kube-api-access-6pzxl\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:03.361464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.361216 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ea9217-3aeb-4591-a754-425c549c29f8-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:03.361464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.361225 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ea9217-3aeb-4591-a754-425c549c29f8-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:03.790532 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.790503 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:33:03.866540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.866448 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ed16d8-1905-44ae-8f12-df385a63dd88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed16d8-1905-44ae-8f12-df385a63dd88-plugins-config-volume\") pod \"a7ed16d8-1905-44ae-8f12-df385a63dd88\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " Nov 04 01:33:03.866718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.866693 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ed16d8-1905-44ae-8f12-df385a63dd88-kube-api-access-tcxsp\" (UniqueName: \"kubernetes.io/projected/a7ed16d8-1905-44ae-8f12-df385a63dd88-kube-api-access-tcxsp\") pod \"a7ed16d8-1905-44ae-8f12-df385a63dd88\" (UID: \"a7ed16d8-1905-44ae-8f12-df385a63dd88\") " Nov 04 01:33:03.866855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.866829 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ed16d8-1905-44ae-8f12-df385a63dd88-plugins-config-volume" pod "a7ed16d8-1905-44ae-8f12-df385a63dd88" (UID: "a7ed16d8-1905-44ae-8f12-df385a63dd88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:03.867143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.867115 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed16d8-1905-44ae-8f12-df385a63dd88-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:03.868806 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.868783 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ed16d8-1905-44ae-8f12-df385a63dd88-kube-api-access-tcxsp" pod "a7ed16d8-1905-44ae-8f12-df385a63dd88" (UID: "a7ed16d8-1905-44ae-8f12-df385a63dd88"). InnerVolumeSpecName "kube-api-access-tcxsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:03.968792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:03.968757 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcxsp\" (UniqueName: \"kubernetes.io/projected/a7ed16d8-1905-44ae-8f12-df385a63dd88-kube-api-access-tcxsp\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:04.109445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.109415 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" path="/var/lib/kubelet/pods/a1d79be2-e8ca-417a-bd4b-aa14b8a237fa/volumes" Nov 04 01:33:04.132589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.132544 4312 generic.go:356] "Generic (PLEG): container finished" podID="a7ed16d8-1905-44ae-8f12-df385a63dd88" containerID="541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c" exitCode=0 Nov 04 01:33:04.133036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.132629 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" event={"ID":"a7ed16d8-1905-44ae-8f12-df385a63dd88","Type":"ContainerDied","Data":"541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c"} Nov 04 01:33:04.133036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.132690 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" event={"ID":"a7ed16d8-1905-44ae-8f12-df385a63dd88","Type":"ContainerDied","Data":"b4000fc77f74475ba9f4cbe74a23f57fd645e4f0a2f20820bb8c9c84591854ba"} Nov 04 01:33:04.133036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.132720 4312 scope.go:122] "RemoveContainer" containerID="541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c" Nov 04 01:33:04.133036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.132733 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v" Nov 04 01:33:04.134800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.134776 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" event={"ID":"44ea9217-3aeb-4591-a754-425c549c29f8","Type":"ContainerDied","Data":"f587996328ccbe8c34f2b5b63d5a9a37b1af0236faaad85d9b8a8f3f5af7a0b7"} Nov 04 01:33:04.134929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.134862 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk" Nov 04 01:33:04.144197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.144181 4312 scope.go:122] "RemoveContainer" containerID="541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c" Nov 04 01:33:04.144492 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:04.144475 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c\": container with ID starting with 541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c not found: ID does not exist" containerID="541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c" Nov 04 01:33:04.144568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.144501 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c"} err="failed to get container status \"541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c\": rpc error: code = NotFound desc = could not find container \"541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c\": container with ID starting with 541248136bb0c4485e1dc73d073424e54e539fdd78f87f224bc21dc0f948196c not found: ID does not exist" Nov 04 01:33:04.144568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.144517 4312 scope.go:122] "RemoveContainer" containerID="eb8664ddf8ce83c894011765e0df3b5620d966cfd31ff4e11711189345984b2b" Nov 04 01:33:04.158257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.158235 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v"] Nov 04 01:33:04.160633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.160610 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8974fa-c797c9f-tlt6v"] Nov 04 01:33:04.165270 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.165245 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" podUID="0c9e25e2-d171-4142-82a6-a877a50cbea6" containerName="payload-processing" containerID="cri-o://c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44" gracePeriod=25 Nov 04 01:33:04.171247 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.171223 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk"] Nov 04 01:33:04.175203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.175179 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aefc5-dbf988499-k62sk"] Nov 04 01:33:04.214174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:04.214141 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" podUID="81e65f77-0ff8-420f-94a6-b029fb7916c9" containerName="payload-pre-processing" containerID="cri-o://5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3" gracePeriod=25 Nov 04 01:33:06.118517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:06.118479 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ea9217-3aeb-4591-a754-425c549c29f8" path="/var/lib/kubelet/pods/44ea9217-3aeb-4591-a754-425c549c29f8/volumes" Nov 04 01:33:06.118918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:06.118904 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ed16d8-1905-44ae-8f12-df385a63dd88" path="/var/lib/kubelet/pods/a7ed16d8-1905-44ae-8f12-df385a63dd88/volumes" Nov 04 01:33:10.574169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:10.574132 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" podUID="0c9e25e2-d171-4142-82a6-a877a50cbea6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:33:10.574595 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:10.574141 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" podUID="81e65f77-0ff8-420f-94a6-b029fb7916c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:33:12.353870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:12.353828 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx"] Nov 04 01:33:12.354406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:12.354082 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" containerID="cri-o://8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700" gracePeriod=30 Nov 04 01:33:14.515902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:14.515866 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv"] Nov 04 01:33:14.535990 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:14.535943 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk"] Nov 04 01:33:17.612148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.612122 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:33:17.641645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.641563 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-certs\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.641645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.641611 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-kube-api-access-vhdsb\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-kube-api-access-vhdsb\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.641852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.641651 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-data\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.641852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.641719 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-token\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-token\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.641852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.641748 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-envoy\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.641852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.641791 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5339c3-a472-44fe-8300-83fdf9114bc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5339c3-a472-44fe-8300-83fdf9114bc5-istiod-ca-cert\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.642077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.641870 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-certs" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:17.642140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642095 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-podinfo\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.642204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642142 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-credential-socket\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.642204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642171 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5339c3-a472-44fe-8300-83fdf9114bc5-istiod-ca-cert" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:17.642306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642240 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-socket\") pod \"6a5339c3-a472-44fe-8300-83fdf9114bc5\" (UID: \"6a5339c3-a472-44fe-8300-83fdf9114bc5\") " Nov 04 01:33:17.642530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642508 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-credential-socket" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:17.642735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642698 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-socket" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:17.642854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642753 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.642854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642772 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.642854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642785 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5339c3-a472-44fe-8300-83fdf9114bc5-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.643020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.642909 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-data" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:17.648480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.648440 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-podinfo" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:17.648480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.648454 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-envoy" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:17.648633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.648463 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-kube-api-access-vhdsb" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "kube-api-access-vhdsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:17.648633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.648484 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-token" pod "6a5339c3-a472-44fe-8300-83fdf9114bc5" (UID: "6a5339c3-a472-44fe-8300-83fdf9114bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:17.744396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.744326 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.744396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.744395 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.744396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.744406 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdsb\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-kube-api-access-vhdsb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.744653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.744417 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.744653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.744426 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.744653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.744434 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5339c3-a472-44fe-8300-83fdf9114bc5-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.833915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.833880 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r"] Nov 04 01:33:17.839779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.839744 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64"] Nov 04 01:33:17.849172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:17.849144 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg"] Nov 04 01:33:18.235287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.235241 4312 generic.go:356] "Generic (PLEG): container finished" podID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerID="8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700" exitCode=0 Nov 04 01:33:18.235534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.235316 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" event={"ID":"6a5339c3-a472-44fe-8300-83fdf9114bc5","Type":"ContainerDied","Data":"8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700"} Nov 04 01:33:18.235534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.235383 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" event={"ID":"6a5339c3-a472-44fe-8300-83fdf9114bc5","Type":"ContainerDied","Data":"93691ea89be392d1729c4d9ae33e172f3e09ae449c1b33e02f3fb88c7d5f2302"} Nov 04 01:33:18.235534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.235415 4312 scope.go:122] "RemoveContainer" containerID="8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700" Nov 04 01:33:18.235767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.235587 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx" Nov 04 01:33:18.252687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.252664 4312 scope.go:122] "RemoveContainer" containerID="7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e" Nov 04 01:33:18.263504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.263465 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx"] Nov 04 01:33:18.264592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.264571 4312 scope.go:122] "RemoveContainer" containerID="8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700" Nov 04 01:33:18.265080 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:18.265053 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700\": container with ID starting with 8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700 not found: ID does not exist" containerID="8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700" Nov 04 01:33:18.265189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.265091 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700"} err="failed to get container status \"8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700\": rpc error: code = NotFound desc = could not find container \"8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700\": container with ID starting with 8ea7339b5478a23cde74c1e963e4eb07a5ba51f943d596e448b7466cd9e8d700 not found: ID does not exist" Nov 04 01:33:18.265189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.265116 4312 scope.go:122] "RemoveContainer" containerID="7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e" Nov 04 01:33:18.265446 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:18.265422 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e\": container with ID starting with 7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e not found: ID does not exist" containerID="7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e" Nov 04 01:33:18.265518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.265455 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e"} err="failed to get container status \"7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e\": rpc error: code = NotFound desc = could not find container \"7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e\": container with ID starting with 7ff3fb72172b01039fd96e4c27e65ea96f86bced60da925606bd13824de5b58e not found: ID does not exist" Nov 04 01:33:18.267708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:18.267686 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-35980d-openshift-default-7f874cd567-js5tx"] Nov 04 01:33:19.516956 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.516918 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" podUID="f20b9587-222a-4254-86f5-480e6c4a2feb" containerName="payload-processing" containerID="cri-o://15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7" gracePeriod=25 Nov 04 01:33:19.537206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.537142 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" podUID="579850bb-cc33-4301-96b0-87cfeef3f110" containerName="payload-pre-processing" containerID="cri-o://4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a" gracePeriod=25 Nov 04 01:33:19.825893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.825869 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:33:19.853524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.853499 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:33:19.873571 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.873529 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/579850bb-cc33-4301-96b0-87cfeef3f110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579850bb-cc33-4301-96b0-87cfeef3f110-plugins-config-volume\") pod \"579850bb-cc33-4301-96b0-87cfeef3f110\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " Nov 04 01:33:19.873837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.873596 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579850bb-cc33-4301-96b0-87cfeef3f110-kube-api-access-5qrtm\" (UniqueName: \"kubernetes.io/projected/579850bb-cc33-4301-96b0-87cfeef3f110-kube-api-access-5qrtm\") pod \"579850bb-cc33-4301-96b0-87cfeef3f110\" (UID: \"579850bb-cc33-4301-96b0-87cfeef3f110\") " Nov 04 01:33:19.874562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.873914 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20b9587-222a-4254-86f5-480e6c4a2feb-kube-api-access-6qlnb\" (UniqueName: \"kubernetes.io/projected/f20b9587-222a-4254-86f5-480e6c4a2feb-kube-api-access-6qlnb\") pod \"f20b9587-222a-4254-86f5-480e6c4a2feb\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " Nov 04 01:33:19.874562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.874147 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20b9587-222a-4254-86f5-480e6c4a2feb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20b9587-222a-4254-86f5-480e6c4a2feb-plugins-config-volume\") pod \"f20b9587-222a-4254-86f5-480e6c4a2feb\" (UID: \"f20b9587-222a-4254-86f5-480e6c4a2feb\") " Nov 04 01:33:19.875156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.874904 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579850bb-cc33-4301-96b0-87cfeef3f110-plugins-config-volume" pod "579850bb-cc33-4301-96b0-87cfeef3f110" (UID: "579850bb-cc33-4301-96b0-87cfeef3f110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:19.875156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.874922 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20b9587-222a-4254-86f5-480e6c4a2feb-plugins-config-volume" pod "f20b9587-222a-4254-86f5-480e6c4a2feb" (UID: "f20b9587-222a-4254-86f5-480e6c4a2feb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:19.875156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.875114 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f20b9587-222a-4254-86f5-480e6c4a2feb-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:19.875156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.875140 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579850bb-cc33-4301-96b0-87cfeef3f110-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:19.877429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.877399 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20b9587-222a-4254-86f5-480e6c4a2feb-kube-api-access-6qlnb" pod "f20b9587-222a-4254-86f5-480e6c4a2feb" (UID: "f20b9587-222a-4254-86f5-480e6c4a2feb"). InnerVolumeSpecName "kube-api-access-6qlnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:19.877808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.877787 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579850bb-cc33-4301-96b0-87cfeef3f110-kube-api-access-5qrtm" pod "579850bb-cc33-4301-96b0-87cfeef3f110" (UID: "579850bb-cc33-4301-96b0-87cfeef3f110"). InnerVolumeSpecName "kube-api-access-5qrtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:19.976527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.976494 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qrtm\" (UniqueName: \"kubernetes.io/projected/579850bb-cc33-4301-96b0-87cfeef3f110-kube-api-access-5qrtm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:19.976527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:19.976523 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qlnb\" (UniqueName: \"kubernetes.io/projected/f20b9587-222a-4254-86f5-480e6c4a2feb-kube-api-access-6qlnb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:20.109107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.109065 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" path="/var/lib/kubelet/pods/6a5339c3-a472-44fe-8300-83fdf9114bc5/volumes" Nov 04 01:33:20.249054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.249015 4312 generic.go:356] "Generic (PLEG): container finished" podID="579850bb-cc33-4301-96b0-87cfeef3f110" containerID="4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a" exitCode=0 Nov 04 01:33:20.249234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.249081 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" Nov 04 01:33:20.249234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.249098 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" event={"ID":"579850bb-cc33-4301-96b0-87cfeef3f110","Type":"ContainerDied","Data":"4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a"} Nov 04 01:33:20.249234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.249158 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk" event={"ID":"579850bb-cc33-4301-96b0-87cfeef3f110","Type":"ContainerDied","Data":"33ff8348bebc786378e67328bf3681912357f1dd47a41066933bd45e1a500590"} Nov 04 01:33:20.249234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.249181 4312 scope.go:122] "RemoveContainer" containerID="4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a" Nov 04 01:33:20.250732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.250708 4312 generic.go:356] "Generic (PLEG): container finished" podID="f20b9587-222a-4254-86f5-480e6c4a2feb" containerID="15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7" exitCode=0 Nov 04 01:33:20.250869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.250765 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" Nov 04 01:33:20.250869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.250798 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" event={"ID":"f20b9587-222a-4254-86f5-480e6c4a2feb","Type":"ContainerDied","Data":"15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7"} Nov 04 01:33:20.250987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.250867 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv" event={"ID":"f20b9587-222a-4254-86f5-480e6c4a2feb","Type":"ContainerDied","Data":"e1902906d5aab915f05667ef36a04692682fb426ad70210ce5fa179761cecc1a"} Nov 04 01:33:20.259187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.259172 4312 scope.go:122] "RemoveContainer" containerID="4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a" Nov 04 01:33:20.259481 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:20.259463 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a\": container with ID starting with 4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a not found: ID does not exist" containerID="4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a" Nov 04 01:33:20.259544 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.259488 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a"} err="failed to get container status \"4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a\": rpc error: code = NotFound desc = could not find container \"4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a\": container with ID starting with 4d61f618df53c69f1a8713118a7e1f8c74e7ca71b6c07a5f4ccf53731171843a not found: ID does not exist" Nov 04 01:33:20.259544 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.259506 4312 scope.go:122] "RemoveContainer" containerID="15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7" Nov 04 01:33:20.267793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.267778 4312 scope.go:122] "RemoveContainer" containerID="15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7" Nov 04 01:33:20.268048 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:20.268018 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7\": container with ID starting with 15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7 not found: ID does not exist" containerID="15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7" Nov 04 01:33:20.268169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.268057 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7"} err="failed to get container status \"15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7\": rpc error: code = NotFound desc = could not find container \"15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7\": container with ID starting with 15c4430f87d2cd68d9804ea586f07129828995f6db59c665efa4b84e2e7761c7 not found: ID does not exist" Nov 04 01:33:20.271087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.271066 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv"] Nov 04 01:33:20.274964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.274941 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0741ef-7876c5b5b-6w2qv"] Nov 04 01:33:20.285733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.285712 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk"] Nov 04 01:33:20.290111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.290093 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0741ef-d467c5f59-wkvzk"] Nov 04 01:33:20.574793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.574757 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" podUID="0c9e25e2-d171-4142-82a6-a877a50cbea6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:33:20.575151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:20.574794 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" podUID="81e65f77-0ff8-420f-94a6-b029fb7916c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:33:22.108797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:22.108767 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579850bb-cc33-4301-96b0-87cfeef3f110" path="/var/lib/kubelet/pods/579850bb-cc33-4301-96b0-87cfeef3f110/volumes" Nov 04 01:33:22.109179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:22.109150 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20b9587-222a-4254-86f5-480e6c4a2feb" path="/var/lib/kubelet/pods/f20b9587-222a-4254-86f5-480e6c4a2feb/volumes" Nov 04 01:33:24.466792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.466762 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:33:24.518427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.518400 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:33:24.523362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.522909 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9e25e2-d171-4142-82a6-a877a50cbea6-kube-api-access-mwfgr\" (UniqueName: \"kubernetes.io/projected/0c9e25e2-d171-4142-82a6-a877a50cbea6-kube-api-access-mwfgr\") pod \"0c9e25e2-d171-4142-82a6-a877a50cbea6\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " Nov 04 01:33:24.523362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.523047 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9e25e2-d171-4142-82a6-a877a50cbea6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9e25e2-d171-4142-82a6-a877a50cbea6-plugins-config-volume\") pod \"0c9e25e2-d171-4142-82a6-a877a50cbea6\" (UID: \"0c9e25e2-d171-4142-82a6-a877a50cbea6\") " Nov 04 01:33:24.531419 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.531385 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9e25e2-d171-4142-82a6-a877a50cbea6-plugins-config-volume" pod "0c9e25e2-d171-4142-82a6-a877a50cbea6" (UID: "0c9e25e2-d171-4142-82a6-a877a50cbea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:24.533100 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.533072 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9e25e2-d171-4142-82a6-a877a50cbea6-kube-api-access-mwfgr" pod "0c9e25e2-d171-4142-82a6-a877a50cbea6" (UID: "0c9e25e2-d171-4142-82a6-a877a50cbea6"). InnerVolumeSpecName "kube-api-access-mwfgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:24.625063 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.625018 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e65f77-0ff8-420f-94a6-b029fb7916c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e65f77-0ff8-420f-94a6-b029fb7916c9-plugins-config-volume\") pod \"81e65f77-0ff8-420f-94a6-b029fb7916c9\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " Nov 04 01:33:24.625243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.625087 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e65f77-0ff8-420f-94a6-b029fb7916c9-kube-api-access-dnt8q\" (UniqueName: \"kubernetes.io/projected/81e65f77-0ff8-420f-94a6-b029fb7916c9-kube-api-access-dnt8q\") pod \"81e65f77-0ff8-420f-94a6-b029fb7916c9\" (UID: \"81e65f77-0ff8-420f-94a6-b029fb7916c9\") " Nov 04 01:33:24.625409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.625393 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9e25e2-d171-4142-82a6-a877a50cbea6-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:24.625463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.625410 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwfgr\" (UniqueName: \"kubernetes.io/projected/0c9e25e2-d171-4142-82a6-a877a50cbea6-kube-api-access-mwfgr\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:24.625463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.625402 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e65f77-0ff8-420f-94a6-b029fb7916c9-plugins-config-volume" pod "81e65f77-0ff8-420f-94a6-b029fb7916c9" (UID: "81e65f77-0ff8-420f-94a6-b029fb7916c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:24.627738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.627714 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e65f77-0ff8-420f-94a6-b029fb7916c9-kube-api-access-dnt8q" pod "81e65f77-0ff8-420f-94a6-b029fb7916c9" (UID: "81e65f77-0ff8-420f-94a6-b029fb7916c9"). InnerVolumeSpecName "kube-api-access-dnt8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:24.726959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.726874 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e65f77-0ff8-420f-94a6-b029fb7916c9-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:24.726959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:24.726903 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnt8q\" (UniqueName: \"kubernetes.io/projected/81e65f77-0ff8-420f-94a6-b029fb7916c9-kube-api-access-dnt8q\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:25.287353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.287297 4312 generic.go:356] "Generic (PLEG): container finished" podID="81e65f77-0ff8-420f-94a6-b029fb7916c9" containerID="5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3" exitCode=0 Nov 04 01:33:25.287504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.287372 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" event={"ID":"81e65f77-0ff8-420f-94a6-b029fb7916c9","Type":"ContainerDied","Data":"5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3"} Nov 04 01:33:25.287504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.287433 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" event={"ID":"81e65f77-0ff8-420f-94a6-b029fb7916c9","Type":"ContainerDied","Data":"d12a8e4092bf875bb86c906011234ea7254d76e5ba0c05bb0c2811c945866bc8"} Nov 04 01:33:25.287504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.287384 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7" Nov 04 01:33:25.287504 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.287454 4312 scope.go:122] "RemoveContainer" containerID="5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3" Nov 04 01:33:25.288900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.288867 4312 generic.go:356] "Generic (PLEG): container finished" podID="0c9e25e2-d171-4142-82a6-a877a50cbea6" containerID="c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44" exitCode=0 Nov 04 01:33:25.289015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.288936 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" event={"ID":"0c9e25e2-d171-4142-82a6-a877a50cbea6","Type":"ContainerDied","Data":"c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44"} Nov 04 01:33:25.289015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.288967 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" Nov 04 01:33:25.289015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.288979 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj" event={"ID":"0c9e25e2-d171-4142-82a6-a877a50cbea6","Type":"ContainerDied","Data":"e0ea7cb0bb2670ec1de2e500bd4d3003d5f5e587553591ba1cbc62fab2582c28"} Nov 04 01:33:25.297804 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.297791 4312 scope.go:122] "RemoveContainer" containerID="5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3" Nov 04 01:33:25.298082 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:25.298064 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3\": container with ID starting with 5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3 not found: ID does not exist" containerID="5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3" Nov 04 01:33:25.298151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.298089 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3"} err="failed to get container status \"5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3\": rpc error: code = NotFound desc = could not find container \"5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3\": container with ID starting with 5673e957eddd1d9aedf05d11d773d89e4b8e885dbad8b0713ec6fe311202f9f3 not found: ID does not exist" Nov 04 01:33:25.298151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.298105 4312 scope.go:122] "RemoveContainer" containerID="c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44" Nov 04 01:33:25.306548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.306528 4312 scope.go:122] "RemoveContainer" containerID="c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44" Nov 04 01:33:25.306804 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:25.306783 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44\": container with ID starting with c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44 not found: ID does not exist" containerID="c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44" Nov 04 01:33:25.306862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.306809 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44"} err="failed to get container status \"c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44\": rpc error: code = NotFound desc = could not find container \"c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44\": container with ID starting with c681a535b9e48f632253ed27766a9a92c956dbae0cb7077747fdcb64e86c7f44 not found: ID does not exist" Nov 04 01:33:25.311309 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.311286 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7"] Nov 04 01:33:25.315070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.315047 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aefc5-5c76c795f-t4wm7"] Nov 04 01:33:25.328174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.328155 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj"] Nov 04 01:33:25.332371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:25.332353 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2aefc5-b46b554c4-plwjj"] Nov 04 01:33:26.107735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:26.107699 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9e25e2-d171-4142-82a6-a877a50cbea6" path="/var/lib/kubelet/pods/0c9e25e2-d171-4142-82a6-a877a50cbea6/volumes" Nov 04 01:33:26.108145 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:26.108095 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e65f77-0ff8-420f-94a6-b029fb7916c9" path="/var/lib/kubelet/pods/81e65f77-0ff8-420f-94a6-b029fb7916c9/volumes" Nov 04 01:33:29.457049 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:29.456968 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:33:30.762114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:30.762074 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5"] Nov 04 01:33:30.762598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:30.762411 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" podUID="1729a5d9-9158-4167-a20f-033ed229a473" containerName="maas-api" containerID="cri-o://2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15" gracePeriod=30 Nov 04 01:33:31.029072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.029040 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:33:31.101347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.100937 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1729a5d9-9158-4167-a20f-033ed229a473-kube-api-access-jbtdb\" (UniqueName: \"kubernetes.io/projected/1729a5d9-9158-4167-a20f-033ed229a473-kube-api-access-jbtdb\") pod \"1729a5d9-9158-4167-a20f-033ed229a473\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " Nov 04 01:33:31.101347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.101147 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-metrics-certs\") pod \"1729a5d9-9158-4167-a20f-033ed229a473\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " Nov 04 01:33:31.101347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.101216 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1729a5d9-9158-4167-a20f-033ed229a473-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1729a5d9-9158-4167-a20f-033ed229a473-openshift-service-ca\") pod \"1729a5d9-9158-4167-a20f-033ed229a473\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " Nov 04 01:33:31.101347 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.101274 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-maas-api-tls\") pod \"1729a5d9-9158-4167-a20f-033ed229a473\" (UID: \"1729a5d9-9158-4167-a20f-033ed229a473\") " Nov 04 01:33:31.101718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.101634 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1729a5d9-9158-4167-a20f-033ed229a473-openshift-service-ca" pod "1729a5d9-9158-4167-a20f-033ed229a473" (UID: "1729a5d9-9158-4167-a20f-033ed229a473"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:31.103634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.103586 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1729a5d9-9158-4167-a20f-033ed229a473-kube-api-access-jbtdb" pod "1729a5d9-9158-4167-a20f-033ed229a473" (UID: "1729a5d9-9158-4167-a20f-033ed229a473"). InnerVolumeSpecName "kube-api-access-jbtdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:31.103764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.103656 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-maas-api-tls" pod "1729a5d9-9158-4167-a20f-033ed229a473" (UID: "1729a5d9-9158-4167-a20f-033ed229a473"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:31.106420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.106391 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-metrics-certs" pod "1729a5d9-9158-4167-a20f-033ed229a473" (UID: "1729a5d9-9158-4167-a20f-033ed229a473"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:31.202902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.202866 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.202902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.202900 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1729a5d9-9158-4167-a20f-033ed229a473-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.202902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.202910 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1729a5d9-9158-4167-a20f-033ed229a473-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.203143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.202919 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbtdb\" (UniqueName: \"kubernetes.io/projected/1729a5d9-9158-4167-a20f-033ed229a473-kube-api-access-jbtdb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.324142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.324108 4312 generic.go:356] "Generic (PLEG): container finished" podID="1729a5d9-9158-4167-a20f-033ed229a473" containerID="2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15" exitCode=0 Nov 04 01:33:31.324301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.324157 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" Nov 04 01:33:31.324301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.324188 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" event={"ID":"1729a5d9-9158-4167-a20f-033ed229a473","Type":"ContainerDied","Data":"2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15"} Nov 04 01:33:31.324301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.324246 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5" event={"ID":"1729a5d9-9158-4167-a20f-033ed229a473","Type":"ContainerDied","Data":"10cc118626521a3e41f3b240ab82d0e799ad2a7f8ea34e85d07f2c74806f8466"} Nov 04 01:33:31.324301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.324260 4312 scope.go:122] "RemoveContainer" containerID="2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15" Nov 04 01:33:31.336802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.336774 4312 scope.go:122] "RemoveContainer" containerID="2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15" Nov 04 01:33:31.337129 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:31.337109 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15\": container with ID starting with 2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15 not found: ID does not exist" containerID="2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15" Nov 04 01:33:31.337193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.337140 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15"} err="failed to get container status \"2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15\": rpc error: code = NotFound desc = could not find container \"2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15\": container with ID starting with 2d849ed83c16d20b288d2942b00a86693eec67d0779c4f4fff32b9e6125e5a15 not found: ID does not exist" Nov 04 01:33:31.352598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.352574 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5"] Nov 04 01:33:31.354520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:31.354501 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ce9e8-7fcf4dfbbf-hwwf5"] Nov 04 01:33:32.109198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:32.109164 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1729a5d9-9158-4167-a20f-033ed229a473" path="/var/lib/kubelet/pods/1729a5d9-9158-4167-a20f-033ed229a473/volumes" Nov 04 01:33:32.855410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:32.855370 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" containerID="cri-o://8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733" gracePeriod=45 Nov 04 01:33:32.859680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:32.859651 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" containerID="cri-o://a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5" gracePeriod=45 Nov 04 01:33:32.869244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:32.869213 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" containerID="cri-o://9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f" gracePeriod=45 Nov 04 01:33:32.984178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:32.984134 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused" Nov 04 01:33:33.124410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.124386 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:33:33.135772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.135675 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:33:33.142930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.142902 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:33:33.226580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226545 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-model-cache\") pod \"1d493695-508b-4253-9229-8465d8ed5626\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " Nov 04 01:33:33.226776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226594 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-home\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-home\") pod \"594998cc-00a8-4cdd-b3f8-9e29a915871a\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " Nov 04 01:33:33.226776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226620 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-home\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-home\") pod \"1d493695-508b-4253-9229-8465d8ed5626\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " Nov 04 01:33:33.226776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226664 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-dshm\") pod \"f879b97c-99be-4398-a09b-866e02a3c8b8\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " Nov 04 01:33:33.226776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226701 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-model-cache\") pod \"594998cc-00a8-4cdd-b3f8-9e29a915871a\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " Nov 04 01:33:33.226776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226763 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-home\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-home\") pod \"f879b97c-99be-4398-a09b-866e02a3c8b8\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226793 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-tmp-dir\") pod \"1d493695-508b-4253-9229-8465d8ed5626\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226831 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-dshm\") pod \"594998cc-00a8-4cdd-b3f8-9e29a915871a\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226859 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594998cc-00a8-4cdd-b3f8-9e29a915871a-kube-api-access-xg77q\" (UniqueName: \"kubernetes.io/projected/594998cc-00a8-4cdd-b3f8-9e29a915871a-kube-api-access-xg77q\") pod \"594998cc-00a8-4cdd-b3f8-9e29a915871a\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226888 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f879b97c-99be-4398-a09b-866e02a3c8b8-kube-api-access-zvhrf\" (UniqueName: \"kubernetes.io/projected/f879b97c-99be-4398-a09b-866e02a3c8b8-kube-api-access-zvhrf\") pod \"f879b97c-99be-4398-a09b-866e02a3c8b8\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226928 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-home" pod "594998cc-00a8-4cdd-b3f8-9e29a915871a" (UID: "594998cc-00a8-4cdd-b3f8-9e29a915871a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226944 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d493695-508b-4253-9229-8465d8ed5626-kube-api-access-bj8bt\" (UniqueName: \"kubernetes.io/projected/1d493695-508b-4253-9229-8465d8ed5626-kube-api-access-bj8bt\") pod \"1d493695-508b-4253-9229-8465d8ed5626\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226947 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-home" pod "1d493695-508b-4253-9229-8465d8ed5626" (UID: "1d493695-508b-4253-9229-8465d8ed5626"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.226978 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f879b97c-99be-4398-a09b-866e02a3c8b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f879b97c-99be-4398-a09b-866e02a3c8b8-tls-certs\") pod \"f879b97c-99be-4398-a09b-866e02a3c8b8\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " Nov 04 01:33:33.227061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227030 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-tmp-dir\") pod \"594998cc-00a8-4cdd-b3f8-9e29a915871a\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " Nov 04 01:33:33.227687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227065 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-tmp-dir\") pod \"f879b97c-99be-4398-a09b-866e02a3c8b8\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " Nov 04 01:33:33.227687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227100 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-dshm\") pod \"1d493695-508b-4253-9229-8465d8ed5626\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " Nov 04 01:33:33.227687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227161 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-tmp-dir" pod "1d493695-508b-4253-9229-8465d8ed5626" (UID: "1d493695-508b-4253-9229-8465d8ed5626"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.227687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227187 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-model-cache\") pod \"f879b97c-99be-4398-a09b-866e02a3c8b8\" (UID: \"f879b97c-99be-4398-a09b-866e02a3c8b8\") " Nov 04 01:33:33.227687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227246 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594998cc-00a8-4cdd-b3f8-9e29a915871a-tls-certs\" (UniqueName: \"kubernetes.io/secret/594998cc-00a8-4cdd-b3f8-9e29a915871a-tls-certs\") pod \"594998cc-00a8-4cdd-b3f8-9e29a915871a\" (UID: \"594998cc-00a8-4cdd-b3f8-9e29a915871a\") " Nov 04 01:33:33.227687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227282 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d493695-508b-4253-9229-8465d8ed5626-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d493695-508b-4253-9229-8465d8ed5626-tls-certs\") pod \"1d493695-508b-4253-9229-8465d8ed5626\" (UID: \"1d493695-508b-4253-9229-8465d8ed5626\") " Nov 04 01:33:33.227687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227382 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-tmp-dir" pod "594998cc-00a8-4cdd-b3f8-9e29a915871a" (UID: "594998cc-00a8-4cdd-b3f8-9e29a915871a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.228165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227759 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-model-cache" pod "1d493695-508b-4253-9229-8465d8ed5626" (UID: "1d493695-508b-4253-9229-8465d8ed5626"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.228165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227958 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.228165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227980 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.228165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.227994 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.228165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.228010 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.228165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.228023 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.228525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.228498 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-home" pod "f879b97c-99be-4398-a09b-866e02a3c8b8" (UID: "f879b97c-99be-4398-a09b-866e02a3c8b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.228735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.228713 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-model-cache" pod "594998cc-00a8-4cdd-b3f8-9e29a915871a" (UID: "594998cc-00a8-4cdd-b3f8-9e29a915871a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.229511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.229008 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-model-cache" pod "f879b97c-99be-4398-a09b-866e02a3c8b8" (UID: "f879b97c-99be-4398-a09b-866e02a3c8b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.229511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.229128 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-tmp-dir" pod "f879b97c-99be-4398-a09b-866e02a3c8b8" (UID: "f879b97c-99be-4398-a09b-866e02a3c8b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.229803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.229749 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-dshm" pod "594998cc-00a8-4cdd-b3f8-9e29a915871a" (UID: "594998cc-00a8-4cdd-b3f8-9e29a915871a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.230763 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.230705 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d493695-508b-4253-9229-8465d8ed5626-tls-certs" pod "1d493695-508b-4253-9229-8465d8ed5626" (UID: "1d493695-508b-4253-9229-8465d8ed5626"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:33.231391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.231255 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d493695-508b-4253-9229-8465d8ed5626-kube-api-access-bj8bt" pod "1d493695-508b-4253-9229-8465d8ed5626" (UID: "1d493695-508b-4253-9229-8465d8ed5626"). InnerVolumeSpecName "kube-api-access-bj8bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.231507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.231474 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594998cc-00a8-4cdd-b3f8-9e29a915871a-tls-certs" pod "594998cc-00a8-4cdd-b3f8-9e29a915871a" (UID: "594998cc-00a8-4cdd-b3f8-9e29a915871a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:33.231694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.231663 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f879b97c-99be-4398-a09b-866e02a3c8b8-kube-api-access-zvhrf" pod "f879b97c-99be-4398-a09b-866e02a3c8b8" (UID: "f879b97c-99be-4398-a09b-866e02a3c8b8"). InnerVolumeSpecName "kube-api-access-zvhrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.231933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.231909 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-dshm" pod "f879b97c-99be-4398-a09b-866e02a3c8b8" (UID: "f879b97c-99be-4398-a09b-866e02a3c8b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.232479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.232455 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-dshm" pod "1d493695-508b-4253-9229-8465d8ed5626" (UID: "1d493695-508b-4253-9229-8465d8ed5626"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.232743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.232724 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594998cc-00a8-4cdd-b3f8-9e29a915871a-kube-api-access-xg77q" pod "594998cc-00a8-4cdd-b3f8-9e29a915871a" (UID: "594998cc-00a8-4cdd-b3f8-9e29a915871a"). InnerVolumeSpecName "kube-api-access-xg77q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.233050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.233033 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f879b97c-99be-4398-a09b-866e02a3c8b8-tls-certs" pod "f879b97c-99be-4398-a09b-866e02a3c8b8" (UID: "f879b97c-99be-4398-a09b-866e02a3c8b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:33.328858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328821 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.328858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328852 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594998cc-00a8-4cdd-b3f8-9e29a915871a-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.328858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328867 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d493695-508b-4253-9229-8465d8ed5626-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328878 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328893 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328904 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328914 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594998cc-00a8-4cdd-b3f8-9e29a915871a-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328926 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg77q\" (UniqueName: \"kubernetes.io/projected/594998cc-00a8-4cdd-b3f8-9e29a915871a-kube-api-access-xg77q\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328941 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhrf\" (UniqueName: \"kubernetes.io/projected/f879b97c-99be-4398-a09b-866e02a3c8b8-kube-api-access-zvhrf\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328953 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj8bt\" (UniqueName: \"kubernetes.io/projected/1d493695-508b-4253-9229-8465d8ed5626-kube-api-access-bj8bt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328965 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f879b97c-99be-4398-a09b-866e02a3c8b8-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328977 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f879b97c-99be-4398-a09b-866e02a3c8b8-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.329114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.328988 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d493695-508b-4253-9229-8465d8ed5626-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.337842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.337813 4312 generic.go:356] "Generic (PLEG): container finished" podID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerID="9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f" exitCode=0 Nov 04 01:33:33.337969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.337854 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" Nov 04 01:33:33.337969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.337899 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" event={"ID":"594998cc-00a8-4cdd-b3f8-9e29a915871a","Type":"ContainerDied","Data":"9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f"} Nov 04 01:33:33.337969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.337943 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg" event={"ID":"594998cc-00a8-4cdd-b3f8-9e29a915871a","Type":"ContainerDied","Data":"24d98aa809cd9f605f23b4d1eee8d8688fc75d0e4d26fc2fd5dba200750f8bfe"} Nov 04 01:33:33.337969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.337967 4312 scope.go:122] "RemoveContainer" containerID="9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f" Nov 04 01:33:33.339290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.339268 4312 generic.go:356] "Generic (PLEG): container finished" podID="1d493695-508b-4253-9229-8465d8ed5626" containerID="8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733" exitCode=0 Nov 04 01:33:33.339435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.339316 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" Nov 04 01:33:33.339435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.339361 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" event={"ID":"1d493695-508b-4253-9229-8465d8ed5626","Type":"ContainerDied","Data":"8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733"} Nov 04 01:33:33.339435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.339397 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r" event={"ID":"1d493695-508b-4253-9229-8465d8ed5626","Type":"ContainerDied","Data":"443b58831420cd2adfba7f0c7a025e9acf0615eeb413829df40f900750dd0479"} Nov 04 01:33:33.340972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.340951 4312 generic.go:356] "Generic (PLEG): container finished" podID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerID="a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5" exitCode=0 Nov 04 01:33:33.341074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.340985 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" event={"ID":"f879b97c-99be-4398-a09b-866e02a3c8b8","Type":"ContainerDied","Data":"a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5"} Nov 04 01:33:33.341074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.341005 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" event={"ID":"f879b97c-99be-4398-a09b-866e02a3c8b8","Type":"ContainerDied","Data":"cbaaef04d8ea85029f6934a702cf2677b9d90c43063d5e0803cd99f0f08de4b8"} Nov 04 01:33:33.341074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.341058 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64" Nov 04 01:33:33.348306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.348286 4312 scope.go:122] "RemoveContainer" containerID="9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f" Nov 04 01:33:33.348642 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:33.348624 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f\": container with ID starting with 9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f not found: ID does not exist" containerID="9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f" Nov 04 01:33:33.348701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.348650 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f"} err="failed to get container status \"9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f\": rpc error: code = NotFound desc = could not find container \"9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f\": container with ID starting with 9ea078f5ccaac244d7ae61758e45aca2b97b1378d5f19ddf727bf57e485fa58f not found: ID does not exist" Nov 04 01:33:33.348701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.348666 4312 scope.go:122] "RemoveContainer" containerID="8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733" Nov 04 01:33:33.358321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.358304 4312 scope.go:122] "RemoveContainer" containerID="8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733" Nov 04 01:33:33.358619 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:33.358600 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733\": container with ID starting with 8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733 not found: ID does not exist" containerID="8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733" Nov 04 01:33:33.358672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.358626 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733"} err="failed to get container status \"8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733\": rpc error: code = NotFound desc = could not find container \"8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733\": container with ID starting with 8673776a7a630bdee4f372e4b3f3f70ff73038f3adbbd57351988189f6c9d733 not found: ID does not exist" Nov 04 01:33:33.358672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.358643 4312 scope.go:122] "RemoveContainer" containerID="a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5" Nov 04 01:33:33.366265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.366247 4312 scope.go:122] "RemoveContainer" containerID="a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5" Nov 04 01:33:33.366554 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:33.366538 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5\": container with ID starting with a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5 not found: ID does not exist" containerID="a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5" Nov 04 01:33:33.366626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.366558 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5"} err="failed to get container status \"a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5\": rpc error: code = NotFound desc = could not find container \"a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5\": container with ID starting with a4a5f9e57d3d759873a18d009ed7693051e274e3c04bd8f96106766c5a937ed5 not found: ID does not exist" Nov 04 01:33:33.369679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.369654 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg"] Nov 04 01:33:33.373566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.373546 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/premium-w0-6dac99-kserve-9896df694-r8qgg"] Nov 04 01:33:33.386600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.386578 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64"] Nov 04 01:33:33.389365 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.389318 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-w0-6dac99-kserve-7dcb599d8f-jcd64"] Nov 04 01:33:33.400729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.400706 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r"] Nov 04 01:33:33.404549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:33.404532 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/distinct-2-w0-6dac99-kserve-5d64c5bf5d-2kk7r"] Nov 04 01:33:34.108189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:34.108159 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d493695-508b-4253-9229-8465d8ed5626" path="/var/lib/kubelet/pods/1d493695-508b-4253-9229-8465d8ed5626/volumes" Nov 04 01:33:34.108779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:34.108759 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" path="/var/lib/kubelet/pods/594998cc-00a8-4cdd-b3f8-9e29a915871a/volumes" Nov 04 01:33:34.109164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:34.109151 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" path="/var/lib/kubelet/pods/f879b97c-99be-4398-a09b-866e02a3c8b8/volumes" Nov 04 01:33:38.284999 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.284966 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt"] Nov 04 01:33:38.285495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.285202 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" podUID="371f9c50-6be6-401a-b489-68abffe8bc46" containerName="maas-api" containerID="cri-o://f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256" gracePeriod=30 Nov 04 01:33:38.564373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.564285 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:33:38.683611 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.683573 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371f9c50-6be6-401a-b489-68abffe8bc46-kube-api-access-lhjl6\" (UniqueName: \"kubernetes.io/projected/371f9c50-6be6-401a-b489-68abffe8bc46-kube-api-access-lhjl6\") pod \"371f9c50-6be6-401a-b489-68abffe8bc46\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " Nov 04 01:33:38.683794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.683631 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-maas-api-tls\") pod \"371f9c50-6be6-401a-b489-68abffe8bc46\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " Nov 04 01:33:38.683794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.683663 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371f9c50-6be6-401a-b489-68abffe8bc46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371f9c50-6be6-401a-b489-68abffe8bc46-openshift-service-ca\") pod \"371f9c50-6be6-401a-b489-68abffe8bc46\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " Nov 04 01:33:38.683794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.683769 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-metrics-certs\") pod \"371f9c50-6be6-401a-b489-68abffe8bc46\" (UID: \"371f9c50-6be6-401a-b489-68abffe8bc46\") " Nov 04 01:33:38.684049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.684027 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371f9c50-6be6-401a-b489-68abffe8bc46-openshift-service-ca" pod "371f9c50-6be6-401a-b489-68abffe8bc46" (UID: "371f9c50-6be6-401a-b489-68abffe8bc46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:38.684308 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.684279 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371f9c50-6be6-401a-b489-68abffe8bc46-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:38.685971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.685940 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-metrics-certs" pod "371f9c50-6be6-401a-b489-68abffe8bc46" (UID: "371f9c50-6be6-401a-b489-68abffe8bc46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:38.686256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.686235 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-maas-api-tls" pod "371f9c50-6be6-401a-b489-68abffe8bc46" (UID: "371f9c50-6be6-401a-b489-68abffe8bc46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:38.686353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.686249 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371f9c50-6be6-401a-b489-68abffe8bc46-kube-api-access-lhjl6" pod "371f9c50-6be6-401a-b489-68abffe8bc46" (UID: "371f9c50-6be6-401a-b489-68abffe8bc46"). InnerVolumeSpecName "kube-api-access-lhjl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:38.785081 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.785044 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjl6\" (UniqueName: \"kubernetes.io/projected/371f9c50-6be6-401a-b489-68abffe8bc46-kube-api-access-lhjl6\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:38.785081 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.785076 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:38.785081 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:38.785086 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371f9c50-6be6-401a-b489-68abffe8bc46-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:39.377306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.377268 4312 generic.go:356] "Generic (PLEG): container finished" podID="371f9c50-6be6-401a-b489-68abffe8bc46" containerID="f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256" exitCode=0 Nov 04 01:33:39.377761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.377317 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" Nov 04 01:33:39.377761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.377364 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" event={"ID":"371f9c50-6be6-401a-b489-68abffe8bc46","Type":"ContainerDied","Data":"f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256"} Nov 04 01:33:39.377761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.377426 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt" event={"ID":"371f9c50-6be6-401a-b489-68abffe8bc46","Type":"ContainerDied","Data":"166f00174b3bbae583729b26a5fa441d1afde3e85e29ec3a4328eb16814f01d2"} Nov 04 01:33:39.377761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.377455 4312 scope.go:122] "RemoveContainer" containerID="f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256" Nov 04 01:33:39.387441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.387413 4312 scope.go:122] "RemoveContainer" containerID="f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256" Nov 04 01:33:39.387804 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:39.387778 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256\": container with ID starting with f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256 not found: ID does not exist" containerID="f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256" Nov 04 01:33:39.387897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.387815 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256"} err="failed to get container status \"f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256\": rpc error: code = NotFound desc = could not find container \"f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256\": container with ID starting with f9497ad6fe3ea014dc1acfccdcf3a0740320a6f380b65d6bd9861c703dc00256 not found: ID does not exist" Nov 04 01:33:39.389634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.389613 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z"] Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390642 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e130b444-8588-48ad-b422-54342eeae735" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390687 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e130b444-8588-48ad-b422-54342eeae735" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390709 4312 cpu_manager.go:414] "containerMap: removing container" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390718 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390752 4312 cpu_manager.go:414] "containerMap: removing container" podUID="757cedc8-09f9-4db9-9a99-72b42341e1f1" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390761 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757cedc8-09f9-4db9-9a99-72b42341e1f1" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390788 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390796 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390824 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f20b9587-222a-4254-86f5-480e6c4a2feb" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390832 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20b9587-222a-4254-86f5-480e6c4a2feb" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390852 4312 cpu_manager.go:414] "containerMap: removing container" podUID="579850bb-cc33-4301-96b0-87cfeef3f110" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390860 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579850bb-cc33-4301-96b0-87cfeef3f110" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390882 4312 cpu_manager.go:414] "containerMap: removing container" podUID="81e65f77-0ff8-420f-94a6-b029fb7916c9" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390889 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e65f77-0ff8-420f-94a6-b029fb7916c9" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390903 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f5b07a71-14ba-435f-bf23-c55c6915c384" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390910 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b07a71-14ba-435f-bf23-c55c6915c384" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390920 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390929 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390955 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390971 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390983 4312 cpu_manager.go:414] "containerMap: removing container" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.390991 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391011 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391018 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391033 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0c9e25e2-d171-4142-82a6-a877a50cbea6" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391040 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9e25e2-d171-4142-82a6-a877a50cbea6" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391069 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391077 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391097 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391104 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391124 4312 cpu_manager.go:414] "containerMap: removing container" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391132 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391148 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391155 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391175 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391183 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391196 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fd92106e-d547-4052-a6a0-de9222fa375a" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391203 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd92106e-d547-4052-a6a0-de9222fa375a" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391220 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391228 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391240 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a7ed16d8-1905-44ae-8f12-df385a63dd88" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391248 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ed16d8-1905-44ae-8f12-df385a63dd88" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391266 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391273 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391302 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8733aede-0f22-4ebe-90e5-636a7409a56d" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391309 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8733aede-0f22-4ebe-90e5-636a7409a56d" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391327 4312 cpu_manager.go:414] "containerMap: removing container" podUID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391462 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391474 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1729a5d9-9158-4167-a20f-033ed229a473" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391482 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1729a5d9-9158-4167-a20f-033ed229a473" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391504 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391511 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391529 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391537 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391557 4312 cpu_manager.go:414] "containerMap: removing container" podUID="44ea9217-3aeb-4591-a754-425c549c29f8" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391565 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ea9217-3aeb-4591-a754-425c549c29f8" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391586 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391594 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391616 4312 cpu_manager.go:414] "containerMap: removing container" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391627 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391647 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391654 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391667 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391674 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391691 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391698 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391708 4312 cpu_manager.go:414] "containerMap: removing container" podUID="20df1fd2-d5ca-4a9a-8280-fed7277e18ac" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391723 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20df1fd2-d5ca-4a9a-8280-fed7277e18ac" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391735 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9a85f039-b6f3-4686-9529-115a20f63093" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391743 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a85f039-b6f3-4686-9529-115a20f63093" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391771 4312 cpu_manager.go:414] "containerMap: removing container" podUID="371f9c50-6be6-401a-b489-68abffe8bc46" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.391778 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371f9c50-6be6-401a-b489-68abffe8bc46" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392040 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392057 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd92106e-d547-4052-a6a0-de9222fa375a" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392083 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392092 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ed16d8-1905-44ae-8f12-df385a63dd88" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392108 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d2a781-cc39-40f3-b2fe-cd5de79d89ff" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392124 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8733aede-0f22-4ebe-90e5-636a7409a56d" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392140 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392148 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="236a8c22-110b-4dd3-a3b0-3687494fd1dc" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392160 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1729a5d9-9158-4167-a20f-033ed229a473" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392176 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392186 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4204c8e6-9476-4a48-a947-635f16cb21c7" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392204 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ea9217-3aeb-4591-a754-425c549c29f8" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392216 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="594998cc-00a8-4cdd-b3f8-9e29a915871a" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392232 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392243 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5339c3-a472-44fe-8300-83fdf9114bc5" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392259 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f879b97c-99be-4398-a09b-866e02a3c8b8" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392268 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="20df1fd2-d5ca-4a9a-8280-fed7277e18ac" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392283 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a85f039-b6f3-4686-9529-115a20f63093" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392299 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="371f9c50-6be6-401a-b489-68abffe8bc46" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392312 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e130b444-8588-48ad-b422-54342eeae735" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392320 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1cbc30-9a61-4274-b2a4-9da61d518423" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392368 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="757cedc8-09f9-4db9-9a99-72b42341e1f1" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392378 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392395 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20b9587-222a-4254-86f5-480e6c4a2feb" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392405 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="579850bb-cc33-4301-96b0-87cfeef3f110" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392423 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e65f77-0ff8-420f-94a6-b029fb7916c9" containerName="payload-pre-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392441 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b07a71-14ba-435f-bf23-c55c6915c384" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392450 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b46565c-06b6-43ea-8d37-da745354cad9" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392466 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc1c7f0-aa3e-4dd9-a58c-14795e9411b6" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392476 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392491 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392502 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9e25e2-d171-4142-82a6-a877a50cbea6" containerName="payload-processing" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392518 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392527 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d493695-508b-4253-9229-8465d8ed5626" containerName="main" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392535 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9beada2e-b29f-4762-8e9d-fd99c4a4fe0c" containerName="maas-api" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392553 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="d807e2eb-432b-4219-814d-d5bbd21a4c53" containerName="istio-proxy" Nov 04 01:33:39.392943 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.392571 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bd26f2-2287-4ae2-9640-dfbe6e1409ee" containerName="payload-processing" Nov 04 01:33:39.405922 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.405890 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt"] Nov 04 01:33:39.406862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.405934 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z"] Nov 04 01:33:39.406862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.406084 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.408791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.408760 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6dac99-557c69646c-6h5qt"] Nov 04 01:33:39.409992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.409970 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:33:39.410693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.410674 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:33:39.411191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.411166 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:33:39.411312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.411268 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:33:39.593578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.593543 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.593749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.593607 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.593749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.593655 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lphnn\" (UniqueName: \"kubernetes.io/projected/a03b246d-c8b8-458f-9fd6-c94f874bae32-kube-api-access-lphnn\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.695108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.695010 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.695108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.695099 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.695369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.695126 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lphnn\" (UniqueName: \"kubernetes.io/projected/a03b246d-c8b8-458f-9fd6-c94f874bae32-kube-api-access-lphnn\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.695781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.695759 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.697435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.697412 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.707361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.707322 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lphnn\" (UniqueName: \"kubernetes.io/projected/a03b246d-c8b8-458f-9fd6-c94f874bae32-kube-api-access-lphnn\") pod \"maas-api-revoke-sub-11809375-c9p4z\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:39.725019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:39.724995 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:40.008695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:40.008669 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z"] Nov 04 01:33:40.010587 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:33:40.010552 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03b246d_c8b8_458f_9fd6_c94f874bae32.slice/crio-73dfb5089ecafe3ee14da2435b9f562f3aa9d626e9410db7c8e8da9c2a22b914 WatchSource:0}: Error finding container 73dfb5089ecafe3ee14da2435b9f562f3aa9d626e9410db7c8e8da9c2a22b914: Status 404 returned error can't find the container with id 73dfb5089ecafe3ee14da2435b9f562f3aa9d626e9410db7c8e8da9c2a22b914 Nov 04 01:33:40.107253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:40.107217 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371f9c50-6be6-401a-b489-68abffe8bc46" path="/var/lib/kubelet/pods/371f9c50-6be6-401a-b489-68abffe8bc46/volumes" Nov 04 01:33:40.383703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:40.383671 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" event={"ID":"a03b246d-c8b8-458f-9fd6-c94f874bae32","Type":"ContainerStarted","Data":"539db97a4031bf788ba54c5fe16a84d0c921e7a233443a9f030a7cee950c74e0"} Nov 04 01:33:40.384177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:40.383727 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" event={"ID":"a03b246d-c8b8-458f-9fd6-c94f874bae32","Type":"ContainerStarted","Data":"73dfb5089ecafe3ee14da2435b9f562f3aa9d626e9410db7c8e8da9c2a22b914"} Nov 04 01:33:40.403717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:40.403671 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" podStartSLOduration=1.403658438 podStartE2EDuration="1.403658438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:40.401503857 +0000 UTC m=+1852.424435880" watchObservedRunningTime="2026-11-04 01:33:40.403658438 +0000 UTC m=+1852.426590444" Nov 04 01:33:41.391120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:41.391085 4312 generic.go:356] "Generic (PLEG): container finished" podID="a03b246d-c8b8-458f-9fd6-c94f874bae32" containerID="539db97a4031bf788ba54c5fe16a84d0c921e7a233443a9f030a7cee950c74e0" exitCode=0 Nov 04 01:33:41.391550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:41.391182 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" event={"ID":"a03b246d-c8b8-458f-9fd6-c94f874bae32","Type":"ContainerDied","Data":"539db97a4031bf788ba54c5fe16a84d0c921e7a233443a9f030a7cee950c74e0"} Nov 04 01:33:42.532057 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.532029 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:42.620817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.620780 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03b246d-c8b8-458f-9fd6-c94f874bae32-kube-api-access-lphnn\" (UniqueName: \"kubernetes.io/projected/a03b246d-c8b8-458f-9fd6-c94f874bae32-kube-api-access-lphnn\") pod \"a03b246d-c8b8-458f-9fd6-c94f874bae32\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " Nov 04 01:33:42.621013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.620875 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-xks-ca\") pod \"a03b246d-c8b8-458f-9fd6-c94f874bae32\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " Nov 04 01:33:42.621013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.620921 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-openshift-ca\") pod \"a03b246d-c8b8-458f-9fd6-c94f874bae32\" (UID: \"a03b246d-c8b8-458f-9fd6-c94f874bae32\") " Nov 04 01:33:42.621310 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.621284 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-openshift-ca" pod "a03b246d-c8b8-458f-9fd6-c94f874bae32" (UID: "a03b246d-c8b8-458f-9fd6-c94f874bae32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:42.622882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.622860 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-xks-ca" pod "a03b246d-c8b8-458f-9fd6-c94f874bae32" (UID: "a03b246d-c8b8-458f-9fd6-c94f874bae32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:42.622882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.622866 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03b246d-c8b8-458f-9fd6-c94f874bae32-kube-api-access-lphnn" pod "a03b246d-c8b8-458f-9fd6-c94f874bae32" (UID: "a03b246d-c8b8-458f-9fd6-c94f874bae32"). InnerVolumeSpecName "kube-api-access-lphnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:42.721874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.721800 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lphnn\" (UniqueName: \"kubernetes.io/projected/a03b246d-c8b8-458f-9fd6-c94f874bae32-kube-api-access-lphnn\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:42.721874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.721830 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:42.721874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.721843 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b246d-c8b8-458f-9fd6-c94f874bae32-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:42.820519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.820484 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc"] Nov 04 01:33:42.820948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.820935 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" Nov 04 01:33:42.820992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.820951 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d79be2-e8ca-417a-bd4b-aa14b8a237fa" containerName="istio-proxy" Nov 04 01:33:42.820992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.820970 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" Nov 04 01:33:42.820992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.820980 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b3756c-b4a5-49f2-a3fb-539cbb63c9c4" containerName="istio-proxy" Nov 04 01:33:42.821159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.821003 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a03b246d-c8b8-458f-9fd6-c94f874bae32" containerName="revoke-keys" Nov 04 01:33:42.821159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.821011 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03b246d-c8b8-458f-9fd6-c94f874bae32" containerName="revoke-keys" Nov 04 01:33:42.821159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.821084 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03b246d-c8b8-458f-9fd6-c94f874bae32" containerName="revoke-keys" Nov 04 01:33:42.821159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.821096 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2b5cb2-ef01-4d87-b48a-878b3239c404" containerName="cleanup" Nov 04 01:33:42.824693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.824675 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.829483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.829457 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6dac99-openshift-default-dockercfg-cxxq4\"" Nov 04 01:33:42.836875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.836853 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj"] Nov 04 01:33:42.853306 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.845591 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.854155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.854130 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp"] Nov 04 01:33:42.867641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.859693 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc"] Nov 04 01:33:42.867641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.859861 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.867641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.863390 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj"] Nov 04 01:33:42.867641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.865633 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp"] Nov 04 01:33:42.923017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.922985 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923022 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923045 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923101 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923124 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923149 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923233 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb61045-4b5a-4e7b-81e6-c30c66237671-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923253 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923271 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgk8\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-kube-api-access-gfgk8\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923289 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6df4853-172a-4a91-8854-3fa6edbd3132-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923309 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kldb\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-kube-api-access-7kldb\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923428 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923456 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923522 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923540 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923554 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l745\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-kube-api-access-9l745\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923569 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923656 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923647 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923666 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:42.923814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923731 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923754 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb76a612-7edb-45a2-820b-ff5500051baa-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923770 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923784 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb76a612-7edb-45a2-820b-ff5500051baa-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923847 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:42.923993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923872 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923900 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:42.923993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:42.923926 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.024989 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kldb\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-kube-api-access-7kldb\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025043 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025072 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.025400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025110 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.025400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025137 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025217 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l745\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-kube-api-access-9l745\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025260 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025319 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.025400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025378 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025413 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025448 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb76a612-7edb-45a2-820b-ff5500051baa-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025471 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025497 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb76a612-7edb-45a2-820b-ff5500051baa-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025528 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025530 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025553 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025564 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025589 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025599 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025626 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025636 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025697 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.025750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025731 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025778 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025803 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025830 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025876 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025911 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025916 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb61045-4b5a-4e7b-81e6-c30c66237671-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025977 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.026027 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.026009 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgk8\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-kube-api-access-gfgk8\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.026083 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6df4853-172a-4a91-8854-3fa6edbd3132-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.026511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.026232 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb76a612-7edb-45a2-820b-ff5500051baa-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.027268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.026579 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb61045-4b5a-4e7b-81e6-c30c66237671-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.027268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.025803 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.027268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.026780 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6df4853-172a-4a91-8854-3fa6edbd3132-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.027268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.027028 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.027268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.027249 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.027649 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.027520 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.028236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.027778 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.028236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.028181 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.028492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.028464 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.028612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.028558 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.028683 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.028631 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb76a612-7edb-45a2-820b-ff5500051baa-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.028865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.028846 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.028947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.028926 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.029236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.029209 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.039212 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.039152 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.039316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.039207 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgk8\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-kube-api-access-gfgk8\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.039538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.039520 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.039654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.039637 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l745\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-kube-api-access-9l745\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.039723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.039635 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.039783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.039756 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kldb\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-kube-api-access-7kldb\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.140376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.140348 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:43.176320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.176295 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:43.186362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.186315 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:43.403410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.403378 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" event={"ID":"a03b246d-c8b8-458f-9fd6-c94f874bae32","Type":"ContainerDied","Data":"73dfb5089ecafe3ee14da2435b9f562f3aa9d626e9410db7c8e8da9c2a22b914"} Nov 04 01:33:43.403410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.403400 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z" Nov 04 01:33:43.403651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.403427 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73dfb5089ecafe3ee14da2435b9f562f3aa9d626e9410db7c8e8da9c2a22b914" Nov 04 01:33:43.421260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.421238 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc"] Nov 04 01:33:43.422527 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:33:43.422507 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb61045_4b5a_4e7b_81e6_c30c66237671.slice/crio-d4d889fb42259089c9fdd9bfa2fca8e97630645b1b2b133ef21337bf162a7869 WatchSource:0}: Error finding container d4d889fb42259089c9fdd9bfa2fca8e97630645b1b2b133ef21337bf162a7869: Status 404 returned error can't find the container with id d4d889fb42259089c9fdd9bfa2fca8e97630645b1b2b133ef21337bf162a7869 Nov 04 01:33:43.424670 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.424644 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.424749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.424701 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.424749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.424730 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.474034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.474006 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj"] Nov 04 01:33:43.476090 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:33:43.476065 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb76a612_7edb_45a2_820b_ff5500051baa.slice/crio-e6759931807bda4c4191c50345baadb021a2d7fabbdaac85cfb462d7d95329ec WatchSource:0}: Error finding container e6759931807bda4c4191c50345baadb021a2d7fabbdaac85cfb462d7d95329ec: Status 404 returned error can't find the container with id e6759931807bda4c4191c50345baadb021a2d7fabbdaac85cfb462d7d95329ec Nov 04 01:33:43.478367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.478328 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.478430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.478404 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.478468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.478438 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.497359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.497322 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp"] Nov 04 01:33:43.509627 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:33:43.509604 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6df4853_172a_4a91_8854_3fa6edbd3132.slice/crio-da1e6be47dc71590fa2f418e2e5101700636b97604f603cac47714eeafd02bd9 WatchSource:0}: Error finding container da1e6be47dc71590fa2f418e2e5101700636b97604f603cac47714eeafd02bd9: Status 404 returned error can't find the container with id da1e6be47dc71590fa2f418e2e5101700636b97604f603cac47714eeafd02bd9 Nov 04 01:33:43.512979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.512697 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.512979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.512787 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:43.512979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:43.512832 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:33:44.428488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.428456 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" event={"ID":"bb76a612-7edb-45a2-820b-ff5500051baa","Type":"ContainerStarted","Data":"dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539"} Nov 04 01:33:44.429105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.428985 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" event={"ID":"bb76a612-7edb-45a2-820b-ff5500051baa","Type":"ContainerStarted","Data":"e6759931807bda4c4191c50345baadb021a2d7fabbdaac85cfb462d7d95329ec"} Nov 04 01:33:44.430613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.430590 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" event={"ID":"3bb61045-4b5a-4e7b-81e6-c30c66237671","Type":"ContainerStarted","Data":"5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd"} Nov 04 01:33:44.430718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.430636 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" event={"ID":"3bb61045-4b5a-4e7b-81e6-c30c66237671","Type":"ContainerStarted","Data":"d4d889fb42259089c9fdd9bfa2fca8e97630645b1b2b133ef21337bf162a7869"} Nov 04 01:33:44.432324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.432307 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" event={"ID":"b6df4853-172a-4a91-8854-3fa6edbd3132","Type":"ContainerStarted","Data":"befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27"} Nov 04 01:33:44.432492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.432476 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" event={"ID":"b6df4853-172a-4a91-8854-3fa6edbd3132","Type":"ContainerStarted","Data":"da1e6be47dc71590fa2f418e2e5101700636b97604f603cac47714eeafd02bd9"} Nov 04 01:33:44.471519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.471455 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podStartSLOduration=2.471436052 podStartE2EDuration="2.471436052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:44.455794626 +0000 UTC m=+1856.478726662" watchObservedRunningTime="2026-11-04 01:33:44.471436052 +0000 UTC m=+1856.494368060" Nov 04 01:33:44.512519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:44.512457 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podStartSLOduration=2.5124357059999998 podStartE2EDuration="2.512435706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:44.509822876 +0000 UTC m=+1856.532754883" watchObservedRunningTime="2026-11-04 01:33:44.512435706 +0000 UTC m=+1856.535367715" Nov 04 01:33:45.141618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.141585 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:45.143273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.143245 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 01:33:45.143429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.143298 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 01:33:45.178129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.178095 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:45.178507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.178485 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 01:33:45.178695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.178671 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 01:33:45.190962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.190937 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:45.191238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.191217 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 01:33:45.191433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:45.191410 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 01:33:46.141837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:46.141800 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 01:33:46.142428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:46.142392 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 01:33:46.183375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:46.182442 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 01:33:46.187355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:46.186853 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 01:33:46.193354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:46.188169 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 01:33:46.193354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:46.188587 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 01:33:47.140937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:47.140896 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 01:33:47.141144 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:47.140945 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 01:33:47.180359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:47.179510 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 01:33:47.180359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:47.179570 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 01:33:47.188357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:47.187177 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 01:33:47.188357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:47.187232 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 01:33:48.141299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:48.140842 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 01:33:48.141299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:48.140894 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 01:33:48.178183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:48.177050 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 01:33:48.178183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:48.177113 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 01:33:48.186810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:48.186668 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 01:33:48.186810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:48.186729 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 01:33:49.159359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:49.146704 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 01:33:49.159359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:49.146799 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 01:33:49.195361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:49.192470 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 01:33:49.195361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:49.192535 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 01:33:49.195361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:49.193201 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 01:33:49.195361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:49.193243 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 01:33:50.164087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:50.164039 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 01:33:50.164846 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:50.164319 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 01:33:50.191356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:50.185851 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 01:33:50.191356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:50.185905 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 01:33:50.193567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:50.193541 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 01:33:50.208816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:50.208773 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 01:33:51.077065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.076976 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podStartSLOduration=9.076955745 podStartE2EDuration="9.076955745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:44.567037158 +0000 UTC m=+1856.589969166" watchObservedRunningTime="2026-11-04 01:33:51.076955745 +0000 UTC m=+1863.099887753" Nov 04 01:33:51.080187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.080160 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr"] Nov 04 01:33:51.088254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.088224 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.093651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.093630 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:33:51.095257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.094101 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr"] Nov 04 01:33:51.099858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.099223 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:33:51.099858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.099307 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:33:51.099858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.099400 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:33:51.217685 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.217620 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdn8p\" (UniqueName: \"kubernetes.io/projected/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-kube-api-access-wdn8p\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.218186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.217826 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.218186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.217897 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.319794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.319750 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.320043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.319841 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.320043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.319924 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdn8p\" (UniqueName: \"kubernetes.io/projected/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-kube-api-access-wdn8p\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.320548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.320511 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.322388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.322326 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.333456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.333368 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdn8p\" (UniqueName: \"kubernetes.io/projected/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-kube-api-access-wdn8p\") pod \"maas-api-revoke-sub-191d967d-7vbrr\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.425623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.425320 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:51.727993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.727859 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:51.729215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.729192 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:51.729924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.729901 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:33:51.739757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.739715 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr"] Nov 04 01:33:51.883000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.881711 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:51.883000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.882856 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:51.890072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.890014 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:33:51.914159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.914119 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:51.915672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.915479 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:51.916787 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:51.916761 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:33:52.542487 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:52.542444 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" event={"ID":"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880","Type":"ContainerStarted","Data":"733dc12458c0376dced644cc9cf2b7693582cbf25df6769491feb630aef81003"} Nov 04 01:33:52.542946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:52.542504 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" event={"ID":"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880","Type":"ContainerStarted","Data":"f863efc9665024b85715f55d44c4463fcc38f34e348f45933393ef66af542c8d"} Nov 04 01:33:52.561926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:52.561882 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" podStartSLOduration=1.561870079 podStartE2EDuration="1.561870079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:52.561223616 +0000 UTC m=+1864.584155622" watchObservedRunningTime="2026-11-04 01:33:52.561870079 +0000 UTC m=+1864.584802111" Nov 04 01:33:53.549241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:53.549207 4312 generic.go:356] "Generic (PLEG): container finished" podID="1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" containerID="733dc12458c0376dced644cc9cf2b7693582cbf25df6769491feb630aef81003" exitCode=0 Nov 04 01:33:53.549632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:53.549292 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" event={"ID":"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880","Type":"ContainerDied","Data":"733dc12458c0376dced644cc9cf2b7693582cbf25df6769491feb630aef81003"} Nov 04 01:33:54.691200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.691171 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:54.769374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.768452 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-kube-api-access-wdn8p\" (UniqueName: \"kubernetes.io/projected/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-kube-api-access-wdn8p\") pod \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " Nov 04 01:33:54.769374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.768519 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-xks-ca\") pod \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " Nov 04 01:33:54.769374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.768628 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-openshift-ca\") pod \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\" (UID: \"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880\") " Nov 04 01:33:54.769374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.769312 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-openshift-ca" pod "1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" (UID: "1008bfc8-a10c-4ce0-9fed-2d5f1d5da880"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:54.774392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.773652 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-xks-ca" pod "1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" (UID: "1008bfc8-a10c-4ce0-9fed-2d5f1d5da880"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:54.775851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.775825 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-kube-api-access-wdn8p" pod "1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" (UID: "1008bfc8-a10c-4ce0-9fed-2d5f1d5da880"). InnerVolumeSpecName "kube-api-access-wdn8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:54.869749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.869656 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdn8p\" (UniqueName: \"kubernetes.io/projected/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-kube-api-access-wdn8p\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:54.869749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.869687 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:54.869749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:54.869698 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.560095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:55.560066 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" Nov 04 01:33:55.560284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:55.560071 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr" event={"ID":"1008bfc8-a10c-4ce0-9fed-2d5f1d5da880","Type":"ContainerDied","Data":"f863efc9665024b85715f55d44c4463fcc38f34e348f45933393ef66af542c8d"} Nov 04 01:33:55.560284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:55.560198 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f863efc9665024b85715f55d44c4463fcc38f34e348f45933393ef66af542c8d" Nov 04 01:33:56.155497 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:56.155464 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj"] Nov 04 01:33:56.156022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:56.155727 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" containerID="cri-o://dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539" gracePeriod=30 Nov 04 01:33:56.167304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:56.167278 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc"] Nov 04 01:33:56.167493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:56.167476 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" containerID="cri-o://5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd" gracePeriod=30 Nov 04 01:33:56.170706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:56.170677 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp"] Nov 04 01:33:56.170902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:33:56.170881 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" containerID="cri-o://befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27" gracePeriod=30 Nov 04 01:33:59.463085 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:33:59.463050 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:34:01.433811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.433740 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:34:01.478652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.478630 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:34:01.480710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.480689 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:34:01.540737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540704 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-certs\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.540909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540757 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb76a612-7edb-45a2-820b-ff5500051baa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb76a612-7edb-45a2-820b-ff5500051baa-istio-podinfo\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.540909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540790 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-token\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-token\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.540909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540827 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-data\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.540909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540857 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-token\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.540909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540897 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-podinfo\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.541184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540965 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-kube-api-access-7kldb\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-kube-api-access-7kldb\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.541184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.540999 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-certs" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.541184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541128 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-data" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.541450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541429 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6df4853-172a-4a91-8854-3fa6edbd3132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6df4853-172a-4a91-8854-3fa6edbd3132-istiod-ca-cert\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.541520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541479 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-socket\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.541586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541517 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6df4853-172a-4a91-8854-3fa6edbd3132-istiod-ca-cert" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:01.541586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541525 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-certs\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.541586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541580 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-envoy\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.541757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541640 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-envoy\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.541825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541753 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb61045-4b5a-4e7b-81e6-c30c66237671-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb61045-4b5a-4e7b-81e6-c30c66237671-istiod-ca-cert\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.541825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541817 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-credential-socket\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.541948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541852 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-kube-api-access-gfgk8\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-kube-api-access-gfgk8\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.541948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541886 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb76a612-7edb-45a2-820b-ff5500051baa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb76a612-7edb-45a2-820b-ff5500051baa-istiod-ca-cert\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.541948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541924 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-credential-socket\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.542122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541957 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-kube-api-access-9l745\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-kube-api-access-9l745\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.542122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542010 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-certs\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.542122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542081 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-socket\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.542122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542114 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-credential-socket\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.542358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542143 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-istio-token\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-istio-token\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.542358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542179 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-socket\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.542358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542234 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-data\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.542358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542262 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-podinfo\") pod \"3bb61045-4b5a-4e7b-81e6-c30c66237671\" (UID: \"3bb61045-4b5a-4e7b-81e6-c30c66237671\") " Nov 04 01:34:01.542358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542308 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-envoy\") pod \"bb76a612-7edb-45a2-820b-ff5500051baa\" (UID: \"bb76a612-7edb-45a2-820b-ff5500051baa\") " Nov 04 01:34:01.542655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542374 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-data\") pod \"b6df4853-172a-4a91-8854-3fa6edbd3132\" (UID: \"b6df4853-172a-4a91-8854-3fa6edbd3132\") " Nov 04 01:34:01.542887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542867 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.542970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542892 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.542970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542907 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6df4853-172a-4a91-8854-3fa6edbd3132-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541711 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-certs" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.541937 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-socket" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.542225 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb61045-4b5a-4e7b-81e6-c30c66237671-istiod-ca-cert" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.543163 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-data" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.543314 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-token" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.543355 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-credential-socket" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.543746 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-credential-socket" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.543768 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-data" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.543826 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-socket" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.544075 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-certs" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.544102 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb76a612-7edb-45a2-820b-ff5500051baa-istiod-ca-cert" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:01.544382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.544278 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-credential-socket" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.545167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.544394 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-socket" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.545431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.545399 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-kube-api-access-7kldb" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "kube-api-access-7kldb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:01.545644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.545614 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-token" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:01.545644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.545620 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb76a612-7edb-45a2-820b-ff5500051baa-istio-podinfo" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:01.545842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.545817 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-envoy" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.545924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.545844 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-envoy" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.546049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.546029 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-podinfo" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:01.546229 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.546206 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-podinfo" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:01.546368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.546316 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-kube-api-access-9l745" pod "3bb61045-4b5a-4e7b-81e6-c30c66237671" (UID: "3bb61045-4b5a-4e7b-81e6-c30c66237671"). InnerVolumeSpecName "kube-api-access-9l745". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:01.546722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.546705 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-kube-api-access-gfgk8" pod "b6df4853-172a-4a91-8854-3fa6edbd3132" (UID: "b6df4853-172a-4a91-8854-3fa6edbd3132"). InnerVolumeSpecName "kube-api-access-gfgk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:01.546917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.546898 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-istio-token" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:01.547568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.547547 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-envoy" pod "bb76a612-7edb-45a2-820b-ff5500051baa" (UID: "bb76a612-7edb-45a2-820b-ff5500051baa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:01.591743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.591715 4312 generic.go:356] "Generic (PLEG): container finished" podID="bb76a612-7edb-45a2-820b-ff5500051baa" containerID="dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539" exitCode=0 Nov 04 01:34:01.591887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.591759 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" Nov 04 01:34:01.591887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.591798 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" event={"ID":"bb76a612-7edb-45a2-820b-ff5500051baa","Type":"ContainerDied","Data":"dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539"} Nov 04 01:34:01.591887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.591856 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj" event={"ID":"bb76a612-7edb-45a2-820b-ff5500051baa","Type":"ContainerDied","Data":"e6759931807bda4c4191c50345baadb021a2d7fabbdaac85cfb462d7d95329ec"} Nov 04 01:34:01.591887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.591885 4312 scope.go:122] "RemoveContainer" containerID="dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539" Nov 04 01:34:01.593162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.593144 4312 generic.go:356] "Generic (PLEG): container finished" podID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerID="5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd" exitCode=0 Nov 04 01:34:01.593249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.593189 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" Nov 04 01:34:01.593249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.593189 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" event={"ID":"3bb61045-4b5a-4e7b-81e6-c30c66237671","Type":"ContainerDied","Data":"5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd"} Nov 04 01:34:01.593375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.593247 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc" event={"ID":"3bb61045-4b5a-4e7b-81e6-c30c66237671","Type":"ContainerDied","Data":"d4d889fb42259089c9fdd9bfa2fca8e97630645b1b2b133ef21337bf162a7869"} Nov 04 01:34:01.594820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.594797 4312 generic.go:356] "Generic (PLEG): container finished" podID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerID="befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27" exitCode=0 Nov 04 01:34:01.594919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.594833 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" event={"ID":"b6df4853-172a-4a91-8854-3fa6edbd3132","Type":"ContainerDied","Data":"befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27"} Nov 04 01:34:01.594919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.594873 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" event={"ID":"b6df4853-172a-4a91-8854-3fa6edbd3132","Type":"ContainerDied","Data":"da1e6be47dc71590fa2f418e2e5101700636b97604f603cac47714eeafd02bd9"} Nov 04 01:34:01.595021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.594935 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp" Nov 04 01:34:01.602511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.602492 4312 scope.go:122] "RemoveContainer" containerID="dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539" Nov 04 01:34:01.602845 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:34:01.602791 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539\": container with ID starting with dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539 not found: ID does not exist" containerID="dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539" Nov 04 01:34:01.602845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.602831 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539"} err="failed to get container status \"dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539\": rpc error: code = NotFound desc = could not find container \"dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539\": container with ID starting with dbb3e53dfabb52e0f317c012a6320da47bcaae9e2296bc25905f3e18fa948539 not found: ID does not exist" Nov 04 01:34:01.602974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.602854 4312 scope.go:122] "RemoveContainer" containerID="5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd" Nov 04 01:34:01.612544 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.612526 4312 scope.go:122] "RemoveContainer" containerID="5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd" Nov 04 01:34:01.612808 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:34:01.612789 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd\": container with ID starting with 5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd not found: ID does not exist" containerID="5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd" Nov 04 01:34:01.612884 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.612818 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd"} err="failed to get container status \"5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd\": rpc error: code = NotFound desc = could not find container \"5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd\": container with ID starting with 5d3674ae112e4afc9060caa4e1df006cb1362a1679595a6f2391014e2872a5fd not found: ID does not exist" Nov 04 01:34:01.612884 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.612843 4312 scope.go:122] "RemoveContainer" containerID="befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27" Nov 04 01:34:01.621516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.621497 4312 scope.go:122] "RemoveContainer" containerID="befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27" Nov 04 01:34:01.621804 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:34:01.621786 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27\": container with ID starting with befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27 not found: ID does not exist" containerID="befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27" Nov 04 01:34:01.621864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.621813 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27"} err="failed to get container status \"befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27\": rpc error: code = NotFound desc = could not find container \"befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27\": container with ID starting with befb6a0134efebb2ae98812a0f4c3564cf7090eb1ec7ea97cb0ccef2db7b1a27 not found: ID does not exist" Nov 04 01:34:01.622132 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.622115 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj"] Nov 04 01:34:01.625987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.625964 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-5pdnj"] Nov 04 01:34:01.638502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.638481 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp"] Nov 04 01:34:01.649196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649169 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649198 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649286 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649278 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649292 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649306 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649320 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649365 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649380 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649392 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649404 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb76a612-7edb-45a2-820b-ff5500051baa-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649416 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649429 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649441 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649454 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kldb\" (UniqueName: \"kubernetes.io/projected/bb76a612-7edb-45a2-820b-ff5500051baa-kube-api-access-7kldb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649468 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649480 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb76a612-7edb-45a2-820b-ff5500051baa-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649492 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649504 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649517 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb61045-4b5a-4e7b-81e6-c30c66237671-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649529 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6df4853-172a-4a91-8854-3fa6edbd3132-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649542 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfgk8\" (UniqueName: \"kubernetes.io/projected/b6df4853-172a-4a91-8854-3fa6edbd3132-kube-api-access-gfgk8\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649554 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb76a612-7edb-45a2-820b-ff5500051baa-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649567 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb61045-4b5a-4e7b-81e6-c30c66237671-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.649970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.649580 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l745\" (UniqueName: \"kubernetes.io/projected/3bb61045-4b5a-4e7b-81e6-c30c66237671-kube-api-access-9l745\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:01.650667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.650647 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-p42vp"] Nov 04 01:34:01.661891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.661870 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc"] Nov 04 01:34:01.666423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:01.666399 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jd9jc"] Nov 04 01:34:02.108980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:02.108951 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" path="/var/lib/kubelet/pods/3bb61045-4b5a-4e7b-81e6-c30c66237671/volumes" Nov 04 01:34:02.109406 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:02.109391 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" path="/var/lib/kubelet/pods/b6df4853-172a-4a91-8854-3fa6edbd3132/volumes" Nov 04 01:34:02.109797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:02.109785 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" path="/var/lib/kubelet/pods/bb76a612-7edb-45a2-820b-ff5500051baa/volumes" Nov 04 01:34:12.768233 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768201 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc"] Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768636 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" containerName="revoke-keys" Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768651 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" containerName="revoke-keys" Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768671 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768676 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768690 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768696 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768717 4312 cpu_manager.go:414] "containerMap: removing container" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" Nov 04 01:34:12.768776 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768722 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" Nov 04 01:34:12.769074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768785 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb76a612-7edb-45a2-820b-ff5500051baa" containerName="istio-proxy" Nov 04 01:34:12.769074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768798 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" containerName="revoke-keys" Nov 04 01:34:12.769074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768806 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6df4853-172a-4a91-8854-3fa6edbd3132" containerName="istio-proxy" Nov 04 01:34:12.769074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.768813 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb61045-4b5a-4e7b-81e6-c30c66237671" containerName="istio-proxy" Nov 04 01:34:12.771933 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.771915 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.792897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.792868 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc"] Nov 04 01:34:12.794924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.794902 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz"] Nov 04 01:34:12.798551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.798530 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.808041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.808013 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz"] Nov 04 01:34:12.867471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867434 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867477 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867556 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8169411b-1b47-4081-bb59-b5d4747cc992-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867643 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867698 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867720 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9f7\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-kube-api-access-xr9f7\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867739 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867842 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.867909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.867878 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8169411b-1b47-4081-bb59-b5d4747cc992-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.969378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969321 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.969561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969390 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9f7\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-kube-api-access-xr9f7\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.969561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969421 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.969561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969467 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.969561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969499 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.969561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969527 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.969856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969564 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8169411b-1b47-4081-bb59-b5d4747cc992-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.969856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969606 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.969856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969818 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.970024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969951 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmv2\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-kube-api-access-jqmv2\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.970024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.969987 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970004 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.970205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970033 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970051 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970056 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be25651c-d37a-4588-93a0-b89625331ede-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.970205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970193 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970265 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970291 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.970463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970351 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be25651c-d37a-4588-93a0-b89625331ede-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:12.970463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970410 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8169411b-1b47-4081-bb59-b5d4747cc992-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970423 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8169411b-1b47-4081-bb59-b5d4747cc992-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970527 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.970797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.970780 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.972273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.972251 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.972564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.972548 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8169411b-1b47-4081-bb59-b5d4747cc992-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.981937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.981916 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9f7\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-kube-api-access-xr9f7\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:12.982057 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:12.981992 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:13.071151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071125 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071162 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071194 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071231 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071248 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmv2\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-kube-api-access-jqmv2\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071301 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071594 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071622 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be25651c-d37a-4588-93a0-b89625331ede-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071650 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071600 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071681 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071736 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be25651c-d37a-4588-93a0-b89625331ede-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.071937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.071829 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.072215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.072193 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be25651c-d37a-4588-93a0-b89625331ede-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.073531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.073509 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.073962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.073946 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be25651c-d37a-4588-93a0-b89625331ede-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.083958 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.083932 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmv2\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-kube-api-access-jqmv2\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.084187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.084170 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.088957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.088938 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:13.115803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.115775 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:13.372321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.372297 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc"] Nov 04 01:34:13.374598 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:34:13.374570 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8169411b_1b47_4081_bb59_b5d4747cc992.slice/crio-b2b8caca7e50d58083182f57cd47f1d371827cc127ef5192ab7e07d9751fa09c WatchSource:0}: Error finding container b2b8caca7e50d58083182f57cd47f1d371827cc127ef5192ab7e07d9751fa09c: Status 404 returned error can't find the container with id b2b8caca7e50d58083182f57cd47f1d371827cc127ef5192ab7e07d9751fa09c Nov 04 01:34:13.376764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.376734 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:13.376839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.376799 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:13.376839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.376826 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:13.407580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.407557 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz"] Nov 04 01:34:13.409686 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:34:13.409662 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe25651c_d37a_4588_93a0_b89625331ede.slice/crio-6f0bee514066d2d1f1752c7aac8a8f5b08d0e77eb37942a3819703c61d4ecdfc WatchSource:0}: Error finding container 6f0bee514066d2d1f1752c7aac8a8f5b08d0e77eb37942a3819703c61d4ecdfc: Status 404 returned error can't find the container with id 6f0bee514066d2d1f1752c7aac8a8f5b08d0e77eb37942a3819703c61d4ecdfc Nov 04 01:34:13.411986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.411958 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:13.412088 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.412055 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:13.412155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.412104 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:13.661859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.661774 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" event={"ID":"be25651c-d37a-4588-93a0-b89625331ede","Type":"ContainerStarted","Data":"20f2fa7f3d641d8567419cbe700b3745cc1c8bf4b40a3984c930da67edef3501"} Nov 04 01:34:13.661859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.661833 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" event={"ID":"be25651c-d37a-4588-93a0-b89625331ede","Type":"ContainerStarted","Data":"6f0bee514066d2d1f1752c7aac8a8f5b08d0e77eb37942a3819703c61d4ecdfc"} Nov 04 01:34:13.663678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.663651 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" event={"ID":"8169411b-1b47-4081-bb59-b5d4747cc992","Type":"ContainerStarted","Data":"1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739"} Nov 04 01:34:13.663807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.663694 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" event={"ID":"8169411b-1b47-4081-bb59-b5d4747cc992","Type":"ContainerStarted","Data":"b2b8caca7e50d58083182f57cd47f1d371827cc127ef5192ab7e07d9751fa09c"} Nov 04 01:34:13.689188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.689141 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podStartSLOduration=1.689127312 podStartE2EDuration="1.689127312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:13.686273219 +0000 UTC m=+1885.709205227" watchObservedRunningTime="2026-11-04 01:34:13.689127312 +0000 UTC m=+1885.712059319" Nov 04 01:34:13.711274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:13.711228 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podStartSLOduration=1.711213877 podStartE2EDuration="1.711213877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:13.709044865 +0000 UTC m=+1885.731976902" watchObservedRunningTime="2026-11-04 01:34:13.711213877 +0000 UTC m=+1885.734145884" Nov 04 01:34:14.092957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:14.092725 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:14.102376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:14.099701 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 01:34:14.102376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:14.099757 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 01:34:14.117371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:14.116211 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:14.117562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:14.117408 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 04 01:34:14.117562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:14.117462 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 04 01:34:15.094399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:15.093730 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 01:34:15.094399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:15.093795 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 01:34:15.116835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:15.116722 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 04 01:34:15.116835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:15.116763 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 04 01:34:16.091921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:16.091880 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 01:34:16.092315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:16.092288 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 01:34:16.117070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:16.117036 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 04 01:34:16.117483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:16.117090 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 04 01:34:17.094356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:17.093425 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 01:34:17.094356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:17.093485 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 01:34:17.116676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:17.116643 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 04 01:34:17.116875 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:17.116690 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 04 01:34:18.095361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:18.091120 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 01:34:18.095361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:18.091190 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 01:34:18.116847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:18.116814 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 04 01:34:18.117030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:18.116859 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 04 01:34:19.100360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:19.099418 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 01:34:19.100360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:19.099478 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 01:34:19.116479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:19.116075 4312 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 04 01:34:19.116479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:19.116132 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 04 01:34:20.094569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:20.094535 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:20.094928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:20.094905 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:20.095637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:20.095599 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:20.121082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:20.121050 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:20.121582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:20.121322 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:20.122803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:20.122778 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:22.169604 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.169565 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt"] Nov 04 01:34:22.170145 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.169787 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" containerID="cri-o://2cc50ef75e73434cc37ce3d696cd93ddadff3c30fecaa2543c7a576df3b34aa0" gracePeriod=30 Nov 04 01:34:22.202674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.202643 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc"] Nov 04 01:34:22.204653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.204630 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd"] Nov 04 01:34:22.204858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.204819 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" containerID="cri-o://615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b" gracePeriod=30 Nov 04 01:34:22.206344 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.206309 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz"] Nov 04 01:34:22.747849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.747788 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" containerID="cri-o://20f2fa7f3d641d8567419cbe700b3745cc1c8bf4b40a3984c930da67edef3501" gracePeriod=30 Nov 04 01:34:22.748068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:22.747820 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" containerID="cri-o://1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739" gracePeriod=30 Nov 04 01:34:26.542100 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.542046 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-brrh6"] Nov 04 01:34:26.547918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.547886 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.556892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.556861 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-brrh6"] Nov 04 01:34:26.557790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.557191 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:34:26.557790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.557235 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:34:26.557790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.557265 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gx6cl\"" Nov 04 01:34:26.558054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.557303 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:34:26.657711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.657677 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.657909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.657867 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjh5\" (UniqueName: \"kubernetes.io/projected/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-kube-api-access-8kjh5\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.657953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.657937 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.759374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.759312 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.759577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.759482 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjh5\" (UniqueName: \"kubernetes.io/projected/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-kube-api-access-8kjh5\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.759577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.759537 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.762403 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.762374 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.762548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.762525 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.772149 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.772117 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjh5\" (UniqueName: \"kubernetes.io/projected/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-kube-api-access-8kjh5\") pod \"maas-controller-5bf8c4d6cc-brrh6\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:26.870502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:26.870413 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:27.159784 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.159751 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-brrh6"] Nov 04 01:34:27.163661 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:34:27.163625 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3126b2a1_0d41_49f5_b300_2f2aed08bdd3.slice/crio-d43b1b532e46dd646464424ea65b6c29a61b8ac82b59934be8f856a74ae113b2 WatchSource:0}: Error finding container d43b1b532e46dd646464424ea65b6c29a61b8ac82b59934be8f856a74ae113b2: Status 404 returned error can't find the container with id d43b1b532e46dd646464424ea65b6c29a61b8ac82b59934be8f856a74ae113b2 Nov 04 01:34:27.165547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.165522 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:34:27.756384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.755518 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.833958 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-istio-token\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-istio-token\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834027 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b81726c-bb5a-4055-b699-50c2042131da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b81726c-bb5a-4055-b699-50c2042131da-istiod-ca-cert\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834134 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-kube-api-access-qhkz6\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-kube-api-access-qhkz6\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834184 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-credential-socket\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834227 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-socket\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834280 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-data\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834378 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-envoy\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834408 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-certs\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834449 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b81726c-bb5a-4055-b699-50c2042131da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b81726c-bb5a-4055-b699-50c2042131da-istio-podinfo\") pod \"6b81726c-bb5a-4055-b699-50c2042131da\" (UID: \"6b81726c-bb5a-4055-b699-50c2042131da\") " Nov 04 01:34:27.843929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.834799 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-credential-socket" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.845600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.844858 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b81726c-bb5a-4055-b699-50c2042131da-istiod-ca-cert" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:27.845964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.845783 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-data" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.846131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.846084 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-socket" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.858366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.852014 4312 generic.go:356] "Generic (PLEG): container finished" podID="6b81726c-bb5a-4055-b699-50c2042131da" containerID="615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b" exitCode=0 Nov 04 01:34:27.858366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.852226 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" Nov 04 01:34:27.863709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.858874 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" event={"ID":"6b81726c-bb5a-4055-b699-50c2042131da","Type":"ContainerDied","Data":"615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b"} Nov 04 01:34:27.863709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.858943 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd" event={"ID":"6b81726c-bb5a-4055-b699-50c2042131da","Type":"ContainerDied","Data":"5777223b477d11d776e871d6e00ab4c908375bed6f073047c82fdf6e0bfe6d84"} Nov 04 01:34:27.863709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.858977 4312 scope.go:122] "RemoveContainer" containerID="615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b" Nov 04 01:34:27.911139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.871589 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-certs" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.911139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.888277 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" event={"ID":"3126b2a1-0d41-49f5-b300-2f2aed08bdd3","Type":"ContainerStarted","Data":"d43b1b532e46dd646464424ea65b6c29a61b8ac82b59934be8f856a74ae113b2"} Nov 04 01:34:27.911139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.891936 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp"] Nov 04 01:34:27.911139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.892159 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" containerID="cri-o://d058c2b3ee026a158bc5e83f2dab3d9913176b5e1047d293ae6cebae13faa638" gracePeriod=30 Nov 04 01:34:27.911139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.904065 4312 generic.go:356] "Generic (PLEG): container finished" podID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerID="2cc50ef75e73434cc37ce3d696cd93ddadff3c30fecaa2543c7a576df3b34aa0" exitCode=0 Nov 04 01:34:27.911139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.904179 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" event={"ID":"298f1428-b1b5-40cd-851f-f3de8c0056a4","Type":"ContainerDied","Data":"2cc50ef75e73434cc37ce3d696cd93ddadff3c30fecaa2543c7a576df3b34aa0"} Nov 04 01:34:27.941186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.941130 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b81726c-bb5a-4055-b699-50c2042131da-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.941186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.941161 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.941580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.941520 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.941580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.941542 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.941580 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.941555 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.003009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.974631 4312 scope.go:122] "RemoveContainer" containerID="615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b" Nov 04 01:34:28.003009 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:34:27.983176 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b\": container with ID starting with 615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b not found: ID does not exist" containerID="615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b" Nov 04 01:34:28.003009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.983221 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b"} err="failed to get container status \"615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b\": rpc error: code = NotFound desc = could not find container \"615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b\": container with ID starting with 615172efc7f6fd8f497126eff887251303814a45ae7583fc27bd322ef0e2511b not found: ID does not exist" Nov 04 01:34:28.003009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:27.983251 4312 scope.go:122] "RemoveContainer" containerID="e106d68eb8246c46cf36cca70120137ec48fc7b9c423cc8bdaa1590ae80de9c2" Nov 04 01:34:28.039652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.038014 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-envoy" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.039652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.038183 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-istio-token" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:28.039652 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.038308 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b81726c-bb5a-4055-b699-50c2042131da-istio-podinfo" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:28.042371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.042303 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-kube-api-access-qhkz6" pod "6b81726c-bb5a-4055-b699-50c2042131da" (UID: "6b81726c-bb5a-4055-b699-50c2042131da"). InnerVolumeSpecName "kube-api-access-qhkz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:28.042797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.042778 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b81726c-bb5a-4055-b699-50c2042131da-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.042870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.042801 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b81726c-bb5a-4055-b699-50c2042131da-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.042870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.042812 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.042870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.042820 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhkz6\" (UniqueName: \"kubernetes.io/projected/6b81726c-bb5a-4055-b699-50c2042131da-kube-api-access-qhkz6\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.173407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.163663 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:34:28.247057 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.246019 4312 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-42sbp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:8080/ready\": dial tcp 10.133.0.24:8080: connect: connection refused" start-of-body= Nov 04 01:34:28.247057 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.246077 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.24:8080/ready\": dial tcp 10.133.0.24:8080: connect: connection refused" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.353241 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-certs\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.358551 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-podinfo\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.358500 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-certs" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.359052 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298f1428-b1b5-40cd-851f-f3de8c0056a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f1428-b1b5-40cd-851f-f3de8c0056a4-istiod-ca-cert\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.359091 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-token\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-token\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.360724 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-data\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.366745 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-data" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.366997 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298f1428-b1b5-40cd-851f-f3de8c0056a4-istiod-ca-cert" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.367062 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-credential-socket\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.367401 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-credential-socket" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.367568 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-envoy\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.367643 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-socket\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.368086 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-kube-api-access-x56c2\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-kube-api-access-x56c2\") pod \"298f1428-b1b5-40cd-851f-f3de8c0056a4\" (UID: \"298f1428-b1b5-40cd-851f-f3de8c0056a4\") " Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.379412 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-socket" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.380039 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.380077 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298f1428-b1b5-40cd-851f-f3de8c0056a4-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.380095 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.380110 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.386662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.380126 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.437809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.420553 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd"] Nov 04 01:34:28.437809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.420791 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-kube-api-access-x56c2" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "kube-api-access-x56c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:28.437809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.435526 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-envoy" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.437809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.435709 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-token" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:28.437809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.436231 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-podinfo" pod "298f1428-b1b5-40cd-851f-f3de8c0056a4" (UID: "298f1428-b1b5-40cd-851f-f3de8c0056a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:28.437809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.437650 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-ktdpd"] Nov 04 01:34:28.505359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.486051 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x56c2\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-kube-api-access-x56c2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.505762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.505730 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.521429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.521395 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.521641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.521620 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298f1428-b1b5-40cd-851f-f3de8c0056a4-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.640942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.640911 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:28.737711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.737626 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8169411b-1b47-4081-bb59-b5d4747cc992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8169411b-1b47-4081-bb59-b5d4747cc992-istiod-ca-cert\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.737856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.737738 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8169411b-1b47-4081-bb59-b5d4747cc992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8169411b-1b47-4081-bb59-b5d4747cc992-istio-podinfo\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.737856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.737791 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-socket\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.737856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.737829 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-istio-token\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-istio-token\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.738002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.737878 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-certs\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.738002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.737914 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-credential-socket\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.738002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.737980 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-data\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.738125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.738030 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-kube-api-access-xr9f7\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-kube-api-access-xr9f7\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.738125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.738058 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-envoy\") pod \"8169411b-1b47-4081-bb59-b5d4747cc992\" (UID: \"8169411b-1b47-4081-bb59-b5d4747cc992\") " Nov 04 01:34:28.738512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.738484 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-credential-socket" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.738704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.738686 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-certs" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.738887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.738869 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8169411b-1b47-4081-bb59-b5d4747cc992-istiod-ca-cert" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:28.738968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.738878 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-data" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.740805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.740780 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-istio-token" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:28.741059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.741033 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-socket" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.741264 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.741189 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-envoy" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:28.742978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.742955 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-kube-api-access-xr9f7" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "kube-api-access-xr9f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:28.743089 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.743025 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8169411b-1b47-4081-bb59-b5d4747cc992-istio-podinfo" pod "8169411b-1b47-4081-bb59-b5d4747cc992" (UID: "8169411b-1b47-4081-bb59-b5d4747cc992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:28.842062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842018 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8169411b-1b47-4081-bb59-b5d4747cc992-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842062 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842064 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8169411b-1b47-4081-bb59-b5d4747cc992-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842081 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842095 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842111 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842127 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842142 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842173 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9f7\" (UniqueName: \"kubernetes.io/projected/8169411b-1b47-4081-bb59-b5d4747cc992-kube-api-access-xr9f7\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.842678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.842188 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8169411b-1b47-4081-bb59-b5d4747cc992-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:28.941384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.940185 4312 generic.go:356] "Generic (PLEG): container finished" podID="be25651c-d37a-4588-93a0-b89625331ede" containerID="20f2fa7f3d641d8567419cbe700b3745cc1c8bf4b40a3984c930da67edef3501" exitCode=0 Nov 04 01:34:28.941384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.940293 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" event={"ID":"be25651c-d37a-4588-93a0-b89625331ede","Type":"ContainerDied","Data":"20f2fa7f3d641d8567419cbe700b3745cc1c8bf4b40a3984c930da67edef3501"} Nov 04 01:34:28.943725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.942466 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" Nov 04 01:34:28.943725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.942857 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt" event={"ID":"298f1428-b1b5-40cd-851f-f3de8c0056a4","Type":"ContainerDied","Data":"cf1a9610a103c8681a8d36f2c193728d033905f3e057f5b33f1c550b9c0480de"} Nov 04 01:34:28.943725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.942918 4312 scope.go:122] "RemoveContainer" containerID="2cc50ef75e73434cc37ce3d696cd93ddadff3c30fecaa2543c7a576df3b34aa0" Nov 04 01:34:28.945702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.945607 4312 generic.go:356] "Generic (PLEG): container finished" podID="8169411b-1b47-4081-bb59-b5d4747cc992" containerID="1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739" exitCode=0 Nov 04 01:34:28.945702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.945680 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" Nov 04 01:34:28.946044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.945718 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" event={"ID":"8169411b-1b47-4081-bb59-b5d4747cc992","Type":"ContainerDied","Data":"1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739"} Nov 04 01:34:28.946044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:28.945763 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc" event={"ID":"8169411b-1b47-4081-bb59-b5d4747cc992","Type":"ContainerDied","Data":"b2b8caca7e50d58083182f57cd47f1d371827cc127ef5192ab7e07d9751fa09c"} Nov 04 01:34:29.038913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.038522 4312 generic.go:356] "Generic (PLEG): container finished" podID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerID="d058c2b3ee026a158bc5e83f2dab3d9913176b5e1047d293ae6cebae13faa638" exitCode=0 Nov 04 01:34:29.038913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.038563 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" event={"ID":"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc","Type":"ContainerDied","Data":"d058c2b3ee026a158bc5e83f2dab3d9913176b5e1047d293ae6cebae13faa638"} Nov 04 01:34:29.090390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.087705 4312 scope.go:122] "RemoveContainer" containerID="1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739" Nov 04 01:34:29.095366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.091878 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt"] Nov 04 01:34:29.127058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.112633 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-5gjpt"] Nov 04 01:34:29.142575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.142548 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:29.150130 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.150028 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc"] Nov 04 01:34:29.158529 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.157323 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-cx9mc"] Nov 04 01:34:29.237869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.237759 4312 scope.go:122] "RemoveContainer" containerID="1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739" Nov 04 01:34:29.239533 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:34:29.239055 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739\": container with ID starting with 1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739 not found: ID does not exist" containerID="1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739" Nov 04 01:34:29.239533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.239099 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739"} err="failed to get container status \"1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739\": rpc error: code = NotFound desc = could not find container \"1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739\": container with ID starting with 1b8845eee9ff94db648ad6f1d82c03590c58366126830677a2b814d588eef739 not found: ID does not exist" Nov 04 01:34:29.239533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.239126 4312 scope.go:122] "RemoveContainer" containerID="b8c4188caf23f249b19a0c5394f02a7e41dc14b43c429f24d39eb8475ea23a89" Nov 04 01:34:29.239533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.239142 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.246741 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-envoy\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.246813 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-kube-api-access-jqmv2\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-kube-api-access-jqmv2\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.246912 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-istio-token\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-istio-token\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.246958 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be25651c-d37a-4588-93a0-b89625331ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be25651c-d37a-4588-93a0-b89625331ede-istio-podinfo\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.246994 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-socket\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.247046 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-data\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.247121 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be25651c-d37a-4588-93a0-b89625331ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be25651c-d37a-4588-93a0-b89625331ede-istiod-ca-cert\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.247232 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-credential-socket\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.247280 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-certs\") pod \"be25651c-d37a-4588-93a0-b89625331ede\" (UID: \"be25651c-d37a-4588-93a0-b89625331ede\") " Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.271605 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-socket" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.271835 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-data" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.272140 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be25651c-d37a-4588-93a0-b89625331ede-istiod-ca-cert" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:29.284738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.284637 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-credential-socket" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:29.285769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.285729 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-certs" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:29.304084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.304020 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-envoy" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:29.307805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.307695 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-istio-token" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:29.308208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.308171 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be25651c-d37a-4588-93a0-b89625331ede-istio-podinfo" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:29.308524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.308497 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-kube-api-access-jqmv2" pod "be25651c-d37a-4588-93a0-b89625331ede" (UID: "be25651c-d37a-4588-93a0-b89625331ede"). InnerVolumeSpecName "kube-api-access-jqmv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.348910 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-local-certs\") pod \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349017 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-cacerts\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-cacerts\") pod \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349084 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-ca-configmap\") pod \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349123 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-kube-api-access-spz2j\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-kube-api-access-spz2j\") pod \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349202 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-kubeconfig\") pod \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349290 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-token\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-token\") pod \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349325 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-dns-cert\") pod \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\" (UID: \"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc\") " Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349889 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349914 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmv2\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-kube-api-access-jqmv2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349930 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be25651c-d37a-4588-93a0-b89625331ede-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349948 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be25651c-d37a-4588-93a0-b89625331ede-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349962 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349976 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.349990 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be25651c-d37a-4588-93a0-b89625331ede-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.350005 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.350021 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be25651c-d37a-4588-93a0-b89625331ede-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.350446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.350247 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-ca-configmap" pod "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" (UID: "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:29.353820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.353749 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-dns-cert" pod "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" (UID: "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:29.353820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.353758 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-local-certs" pod "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" (UID: "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:29.354032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.353944 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-cacerts" pod "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" (UID: "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:29.354358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.354305 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-token" pod "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" (UID: "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:29.354616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.354582 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-kube-api-access-spz2j" pod "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" (UID: "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc"). InnerVolumeSpecName "kube-api-access-spz2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:29.355164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.355143 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-kubeconfig" pod "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" (UID: "6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:29.451392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.451320 4312 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-cacerts\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.451392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.451379 4312 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-ca-configmap\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.451392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.451397 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spz2j\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-kube-api-access-spz2j\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.451648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.451413 4312 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-kubeconfig\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.451648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.451430 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.451648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.451445 4312 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-istio-csr-dns-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.451648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:29.451461 4312 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc-local-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.468243 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:34:29.468177 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:34:30.048290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.048256 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" Nov 04 01:34:30.048791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.048243 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz" event={"ID":"be25651c-d37a-4588-93a0-b89625331ede","Type":"ContainerDied","Data":"6f0bee514066d2d1f1752c7aac8a8f5b08d0e77eb37942a3819703c61d4ecdfc"} Nov 04 01:34:30.048791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.048440 4312 scope.go:122] "RemoveContainer" containerID="20f2fa7f3d641d8567419cbe700b3745cc1c8bf4b40a3984c930da67edef3501" Nov 04 01:34:30.052270 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.052218 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" event={"ID":"6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc","Type":"ContainerDied","Data":"93fe92507b545d4c0f6c55f66b6160f61ff49c75537909918804bd07c43888b9"} Nov 04 01:34:30.052270 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.052239 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp" Nov 04 01:34:30.060533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.060194 4312 scope.go:122] "RemoveContainer" containerID="d058c2b3ee026a158bc5e83f2dab3d9913176b5e1047d293ae6cebae13faa638" Nov 04 01:34:30.096126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.096098 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz"] Nov 04 01:34:30.110082 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.110052 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" path="/var/lib/kubelet/pods/298f1428-b1b5-40cd-851f-f3de8c0056a4/volumes" Nov 04 01:34:30.110542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.110526 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b81726c-bb5a-4055-b699-50c2042131da" path="/var/lib/kubelet/pods/6b81726c-bb5a-4055-b699-50c2042131da/volumes" Nov 04 01:34:30.111026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.110997 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" path="/var/lib/kubelet/pods/8169411b-1b47-4081-bb59-b5d4747cc992/volumes" Nov 04 01:34:30.111578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.111562 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-54qdz"] Nov 04 01:34:30.112823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.112801 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp"] Nov 04 01:34:30.116681 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:30.116659 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-42sbp"] Nov 04 01:34:32.066219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:32.066181 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" event={"ID":"3126b2a1-0d41-49f5-b300-2f2aed08bdd3","Type":"ContainerStarted","Data":"fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834"} Nov 04 01:34:32.066737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:32.066289 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:32.089409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:32.089363 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" podStartSLOduration=2.145499135 podStartE2EDuration="6.089351179s" totalImagesPullingTime="3.943852044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:32.086412969 +0000 UTC m=+1904.109344996" watchObservedRunningTime="2026-11-04 01:34:32.089351179 +0000 UTC m=+1904.112283185" Nov 04 01:34:32.106114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:32.106088 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" path="/var/lib/kubelet/pods/6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc/volumes" Nov 04 01:34:32.106543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:32.106529 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be25651c-d37a-4588-93a0-b89625331ede" path="/var/lib/kubelet/pods/be25651c-d37a-4588-93a0-b89625331ede/volumes" Nov 04 01:34:42.889845 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.889807 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv"] Nov 04 01:34:42.890245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890216 4312 cpu_manager.go:414] "containerMap: removing container" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" Nov 04 01:34:42.890245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890229 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890252 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890257 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890270 4312 cpu_manager.go:414] "containerMap: removing container" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890275 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890281 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890286 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890298 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890303 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890311 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" Nov 04 01:34:42.890373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890316 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" Nov 04 01:34:42.890740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890393 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" Nov 04 01:34:42.890740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890404 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b81726c-bb5a-4055-b699-50c2042131da" containerName="istio-proxy" Nov 04 01:34:42.890740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890412 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="be25651c-d37a-4588-93a0-b89625331ede" containerName="istio-proxy" Nov 04 01:34:42.890740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890421 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa7d9ce-8af6-42ac-8eca-e80b5d3ec3fc" containerName="discovery" Nov 04 01:34:42.890740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890428 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" Nov 04 01:34:42.890740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890433 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" Nov 04 01:34:42.890740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.890440 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8169411b-1b47-4081-bb59-b5d4747cc992" containerName="istio-proxy" Nov 04 01:34:42.893387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.893369 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.904811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.904787 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv"] Nov 04 01:34:42.987064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987028 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987064 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987066 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987139 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987165 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73cdf89-4461-4224-8cf1-53353a26afd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987197 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmc6\" (UniqueName: \"kubernetes.io/projected/d73cdf89-4461-4224-8cf1-53353a26afd5-kube-api-access-hpmc6\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987214 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987292 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987419 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:42.987459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:42.987443 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.073705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.073670 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:34:43.088909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.088881 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.088917 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.088957 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.088985 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73cdf89-4461-4224-8cf1-53353a26afd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089028 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmc6\" (UniqueName: \"kubernetes.io/projected/d73cdf89-4461-4224-8cf1-53353a26afd5-kube-api-access-hpmc6\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089055 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089090 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089126 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089153 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089320 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089387 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089701 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089723 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089715 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.089869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.089851 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73cdf89-4461-4224-8cf1-53353a26afd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.091386 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.091368 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.091810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.091791 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.103835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.103813 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmc6\" (UniqueName: \"kubernetes.io/projected/d73cdf89-4461-4224-8cf1-53353a26afd5-kube-api-access-hpmc6\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.104033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.104012 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73cdf89-4461-4224-8cf1-53353a26afd5-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-m8lgv\" (UID: \"d73cdf89-4461-4224-8cf1-53353a26afd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.206666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.206591 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:43.485938 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.485909 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv"] Nov 04 01:34:43.487378 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:34:43.487349 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73cdf89_4461_4224_8cf1_53353a26afd5.slice/crio-fa0e67c0145ca2066c31cf834f5b74c5832730ffec59833a4410272debbb86e4 WatchSource:0}: Error finding container fa0e67c0145ca2066c31cf834f5b74c5832730ffec59833a4410272debbb86e4: Status 404 returned error can't find the container with id fa0e67c0145ca2066c31cf834f5b74c5832730ffec59833a4410272debbb86e4 Nov 04 01:34:43.489399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.489367 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:43.489514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.489441 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:43.489514 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:43.489488 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:34:44.138404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:44.138368 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" event={"ID":"d73cdf89-4461-4224-8cf1-53353a26afd5","Type":"ContainerStarted","Data":"c25d46d08ffeb1e5730fdfd1dede5b61c15235d9c9ba0e1591ddd8b39eac940f"} Nov 04 01:34:44.138976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:44.138426 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" event={"ID":"d73cdf89-4461-4224-8cf1-53353a26afd5","Type":"ContainerStarted","Data":"fa0e67c0145ca2066c31cf834f5b74c5832730ffec59833a4410272debbb86e4"} Nov 04 01:34:44.173426 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:44.173351 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podStartSLOduration=2.173311452 podStartE2EDuration="2.173311452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:44.16279147 +0000 UTC m=+1916.185723478" watchObservedRunningTime="2026-11-04 01:34:44.173311452 +0000 UTC m=+1916.196243460" Nov 04 01:34:44.207495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:44.207468 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:44.208807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:44.208786 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:44.208931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:44.208828 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:45.207664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:45.207631 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:45.208078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:45.207696 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:46.207527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:46.207489 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:46.207745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:46.207546 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:47.207716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:47.207681 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:47.207928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:47.207741 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:48.207246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.207209 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:48.207442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.207277 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:48.522549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.522460 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj"] Nov 04 01:34:48.523093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.523067 4312 cpu_manager.go:414] "containerMap: removing container" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" Nov 04 01:34:48.523093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.523089 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298f1428-b1b5-40cd-851f-f3de8c0056a4" containerName="istio-proxy" Nov 04 01:34:48.526557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.526536 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.543439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.543406 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:34:48.543592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.543481 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:34:48.543592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.543529 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:34:48.543592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.543550 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:34:48.543843 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.543823 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj"] Nov 04 01:34:48.642205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.642169 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.642425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.642259 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.642425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.642393 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgpg9\" (UniqueName: \"kubernetes.io/projected/5d40cf27-a4fb-4057-9b57-d829decae5f9-kube-api-access-vgpg9\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.743817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.743779 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpg9\" (UniqueName: \"kubernetes.io/projected/5d40cf27-a4fb-4057-9b57-d829decae5f9-kube-api-access-vgpg9\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.744027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.743859 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.744027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.743902 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.744558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.744531 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.746213 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.746191 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.775143 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.775067 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpg9\" (UniqueName: \"kubernetes.io/projected/5d40cf27-a4fb-4057-9b57-d829decae5f9-kube-api-access-vgpg9\") pod \"maas-api-revoke-sub-85237e45-2pzqj\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:48.840139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:48.840110 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:49.144211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:49.143073 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj"] Nov 04 01:34:49.146217 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:34:49.146190 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d40cf27_a4fb_4057_9b57_d829decae5f9.slice/crio-b6f33dd3d1fca306954f91880a5d52f1bd505ca93a10784d4fca9a578ca64884 WatchSource:0}: Error finding container b6f33dd3d1fca306954f91880a5d52f1bd505ca93a10784d4fca9a578ca64884: Status 404 returned error can't find the container with id b6f33dd3d1fca306954f91880a5d52f1bd505ca93a10784d4fca9a578ca64884 Nov 04 01:34:49.191371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:49.191317 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" event={"ID":"5d40cf27-a4fb-4057-9b57-d829decae5f9","Type":"ContainerStarted","Data":"b6f33dd3d1fca306954f91880a5d52f1bd505ca93a10784d4fca9a578ca64884"} Nov 04 01:34:49.207379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:49.207355 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:49.207481 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:49.207405 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:50.201023 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:50.200981 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" event={"ID":"5d40cf27-a4fb-4057-9b57-d829decae5f9","Type":"ContainerStarted","Data":"06846f3c142a0fc8147c0ac88612b7d4d513f7fe962d67007ffa5c3c9517513c"} Nov 04 01:34:50.207294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:50.207262 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:50.207498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:50.207319 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:50.222424 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:50.222368 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" podStartSLOduration=2.222350576 podStartE2EDuration="2.222350576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:50.219959251 +0000 UTC m=+1922.242891273" watchObservedRunningTime="2026-11-04 01:34:50.222350576 +0000 UTC m=+1922.245282575" Nov 04 01:34:51.206632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:51.206595 4312 generic.go:356] "Generic (PLEG): container finished" podID="5d40cf27-a4fb-4057-9b57-d829decae5f9" containerID="06846f3c142a0fc8147c0ac88612b7d4d513f7fe962d67007ffa5c3c9517513c" exitCode=0 Nov 04 01:34:51.207049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:51.206638 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" event={"ID":"5d40cf27-a4fb-4057-9b57-d829decae5f9","Type":"ContainerDied","Data":"06846f3c142a0fc8147c0ac88612b7d4d513f7fe962d67007ffa5c3c9517513c"} Nov 04 01:34:51.207049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:51.206878 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 01:34:51.207049 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:51.206914 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 01:34:52.354213 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.354181 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:52.507931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.507832 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-openshift-ca\") pod \"5d40cf27-a4fb-4057-9b57-d829decae5f9\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " Nov 04 01:34:52.507931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.507879 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-xks-ca\") pod \"5d40cf27-a4fb-4057-9b57-d829decae5f9\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " Nov 04 01:34:52.507931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.507917 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d40cf27-a4fb-4057-9b57-d829decae5f9-kube-api-access-vgpg9\" (UniqueName: \"kubernetes.io/projected/5d40cf27-a4fb-4057-9b57-d829decae5f9-kube-api-access-vgpg9\") pod \"5d40cf27-a4fb-4057-9b57-d829decae5f9\" (UID: \"5d40cf27-a4fb-4057-9b57-d829decae5f9\") " Nov 04 01:34:52.508256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.508189 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-openshift-ca" pod "5d40cf27-a4fb-4057-9b57-d829decae5f9" (UID: "5d40cf27-a4fb-4057-9b57-d829decae5f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:52.508326 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.508315 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.509942 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.509915 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-xks-ca" pod "5d40cf27-a4fb-4057-9b57-d829decae5f9" (UID: "5d40cf27-a4fb-4057-9b57-d829decae5f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:52.510078 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.509986 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d40cf27-a4fb-4057-9b57-d829decae5f9-kube-api-access-vgpg9" pod "5d40cf27-a4fb-4057-9b57-d829decae5f9" (UID: "5d40cf27-a4fb-4057-9b57-d829decae5f9"). InnerVolumeSpecName "kube-api-access-vgpg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:52.608997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.608955 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d40cf27-a4fb-4057-9b57-d829decae5f9-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.608997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:52.608996 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgpg9\" (UniqueName: \"kubernetes.io/projected/5d40cf27-a4fb-4057-9b57-d829decae5f9-kube-api-access-vgpg9\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:53.219168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:53.219133 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" event={"ID":"5d40cf27-a4fb-4057-9b57-d829decae5f9","Type":"ContainerDied","Data":"b6f33dd3d1fca306954f91880a5d52f1bd505ca93a10784d4fca9a578ca64884"} Nov 04 01:34:53.219399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:53.219181 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f33dd3d1fca306954f91880a5d52f1bd505ca93a10784d4fca9a578ca64884" Nov 04 01:34:53.219399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:53.219142 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj" Nov 04 01:34:53.249494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:53.249458 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:34:53.249665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:53.249538 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:34:54.291575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:54.291519 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:34:54.292110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:54.291599 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:34:54.481159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:54.481121 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:54.481524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:54.481505 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:54.482252 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:34:54.482230 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" Nov 04 01:34:59.458224 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:34:59.458146 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:35:00.216818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.216780 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-brrh6"] Nov 04 01:35:00.217099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.217036 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" podUID="3126b2a1-0d41-49f5-b300-2f2aed08bdd3" containerName="manager" containerID="cri-o://fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834" gracePeriod=30 Nov 04 01:35:00.469121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.469068 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:35:00.587464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.587430 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-metrics-certs\") pod \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " Nov 04 01:35:00.587464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.587475 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-webhook-cert\") pod \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " Nov 04 01:35:00.587732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.587512 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-kube-api-access-8kjh5\" (UniqueName: \"kubernetes.io/projected/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-kube-api-access-8kjh5\") pod \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\" (UID: \"3126b2a1-0d41-49f5-b300-2f2aed08bdd3\") " Nov 04 01:35:00.589581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.589551 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-webhook-cert" pod "3126b2a1-0d41-49f5-b300-2f2aed08bdd3" (UID: "3126b2a1-0d41-49f5-b300-2f2aed08bdd3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:00.589581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.589574 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-metrics-certs" pod "3126b2a1-0d41-49f5-b300-2f2aed08bdd3" (UID: "3126b2a1-0d41-49f5-b300-2f2aed08bdd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:00.589772 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.589581 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-kube-api-access-8kjh5" pod "3126b2a1-0d41-49f5-b300-2f2aed08bdd3" (UID: "3126b2a1-0d41-49f5-b300-2f2aed08bdd3"). InnerVolumeSpecName "kube-api-access-8kjh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:00.689170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.689145 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:00.689170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.689169 4312 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-webhook-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:00.689370 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:00.689184 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjh5\" (UniqueName: \"kubernetes.io/projected/3126b2a1-0d41-49f5-b300-2f2aed08bdd3-kube-api-access-8kjh5\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:01.263478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.263438 4312 generic.go:356] "Generic (PLEG): container finished" podID="3126b2a1-0d41-49f5-b300-2f2aed08bdd3" containerID="fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834" exitCode=0 Nov 04 01:35:01.263676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.263494 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" Nov 04 01:35:01.263676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.263526 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" event={"ID":"3126b2a1-0d41-49f5-b300-2f2aed08bdd3","Type":"ContainerDied","Data":"fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834"} Nov 04 01:35:01.263676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.263589 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-brrh6" event={"ID":"3126b2a1-0d41-49f5-b300-2f2aed08bdd3","Type":"ContainerDied","Data":"d43b1b532e46dd646464424ea65b6c29a61b8ac82b59934be8f856a74ae113b2"} Nov 04 01:35:01.263676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.263627 4312 scope.go:122] "RemoveContainer" containerID="fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834" Nov 04 01:35:01.273241 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.273223 4312 scope.go:122] "RemoveContainer" containerID="fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834" Nov 04 01:35:01.273589 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:35:01.273568 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834\": container with ID starting with fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834 not found: ID does not exist" containerID="fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834" Nov 04 01:35:01.273648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.273598 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834"} err="failed to get container status \"fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834\": rpc error: code = NotFound desc = could not find container \"fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834\": container with ID starting with fb1bd48f6c2f26d6d4f31f98664705722b86ab5a7b15dac216de641e4a508834 not found: ID does not exist" Nov 04 01:35:01.286545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.286518 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-brrh6"] Nov 04 01:35:01.290666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:01.290640 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-brrh6"] Nov 04 01:35:02.107244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.107214 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3126b2a1-0d41-49f5-b300-2f2aed08bdd3" path="/var/lib/kubelet/pods/3126b2a1-0d41-49f5-b300-2f2aed08bdd3/volumes" Nov 04 01:35:02.513528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.513442 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-vmh4f"] Nov 04 01:35:02.513931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.513912 4312 cpu_manager.go:414] "containerMap: removing container" podUID="5d40cf27-a4fb-4057-9b57-d829decae5f9" containerName="revoke-keys" Nov 04 01:35:02.514035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.513933 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d40cf27-a4fb-4057-9b57-d829decae5f9" containerName="revoke-keys" Nov 04 01:35:02.514035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.513975 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3126b2a1-0d41-49f5-b300-2f2aed08bdd3" containerName="manager" Nov 04 01:35:02.514035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.513984 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3126b2a1-0d41-49f5-b300-2f2aed08bdd3" containerName="manager" Nov 04 01:35:02.514191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.514099 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3126b2a1-0d41-49f5-b300-2f2aed08bdd3" containerName="manager" Nov 04 01:35:02.514191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.514119 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d40cf27-a4fb-4057-9b57-d829decae5f9" containerName="revoke-keys" Nov 04 01:35:02.518512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.518490 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.523589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.523572 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:35:02.523687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.523572 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:35:02.523687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.523600 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gx6cl\"" Nov 04 01:35:02.525727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.525372 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:35:02.529178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.529152 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-vmh4f"] Nov 04 01:35:02.606839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.606807 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a14699c-70df-4f2c-850d-ee52d69d89de-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.607016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.606855 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a14699c-70df-4f2c-850d-ee52d69d89de-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.607016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.606889 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cz9\" (UniqueName: \"kubernetes.io/projected/0a14699c-70df-4f2c-850d-ee52d69d89de-kube-api-access-f9cz9\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.708186 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.708152 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a14699c-70df-4f2c-850d-ee52d69d89de-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.708402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.708205 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cz9\" (UniqueName: \"kubernetes.io/projected/0a14699c-70df-4f2c-850d-ee52d69d89de-kube-api-access-f9cz9\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.708402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.708362 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a14699c-70df-4f2c-850d-ee52d69d89de-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.710708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.710681 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a14699c-70df-4f2c-850d-ee52d69d89de-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.710797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.710743 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a14699c-70df-4f2c-850d-ee52d69d89de-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.721055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.721035 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cz9\" (UniqueName: \"kubernetes.io/projected/0a14699c-70df-4f2c-850d-ee52d69d89de-kube-api-access-f9cz9\") pod \"maas-controller-5bf8c4d6cc-vmh4f\" (UID: \"0a14699c-70df-4f2c-850d-ee52d69d89de\") " pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:02.832814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:02.832781 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:03.124096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:03.124068 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-vmh4f"] Nov 04 01:35:03.125050 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:35:03.125025 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a14699c_70df_4f2c_850d_ee52d69d89de.slice/crio-9a8746c31d8fb614542f1e5590961eea0642f4f969cd7ca9601b44b5c42c9e99 WatchSource:0}: Error finding container 9a8746c31d8fb614542f1e5590961eea0642f4f969cd7ca9601b44b5c42c9e99: Status 404 returned error can't find the container with id 9a8746c31d8fb614542f1e5590961eea0642f4f969cd7ca9601b44b5c42c9e99 Nov 04 01:35:03.274895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:03.274860 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" event={"ID":"0a14699c-70df-4f2c-850d-ee52d69d89de","Type":"ContainerStarted","Data":"9a8746c31d8fb614542f1e5590961eea0642f4f969cd7ca9601b44b5c42c9e99"} Nov 04 01:35:04.281470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:04.281431 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" event={"ID":"0a14699c-70df-4f2c-850d-ee52d69d89de","Type":"ContainerStarted","Data":"2549c6c72aa2827d1b1d94baefa120075109888f0d27217567131a0f532cadfc"} Nov 04 01:35:04.281972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:04.281534 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:04.304609 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:04.304557 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" podStartSLOduration=1.846824728 podStartE2EDuration="2.304541811s" totalImagesPullingTime="457.717083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:04.302378921 +0000 UTC m=+1936.325310934" watchObservedRunningTime="2026-11-04 01:35:04.304541811 +0000 UTC m=+1936.327473818" Nov 04 01:35:15.289136 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:15.289104 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bf8c4d6cc-vmh4f" Nov 04 01:35:29.458240 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:35:29.458187 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:35:40.526916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:40.526477 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:40.526916 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:40.526559 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:42.198369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.198316 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw"] Nov 04 01:35:42.201995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.201975 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.207880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.207845 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:42.207880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.207855 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:42.208613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.208589 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:42.208780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.208645 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:35:42.212172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.212148 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw"] Nov 04 01:35:42.245005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.244972 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqwq\" (UniqueName: \"kubernetes.io/projected/83540a58-81a7-41de-b896-cb38b0b0e3ce-kube-api-access-lbqwq\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.245199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.245018 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.245199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.245118 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.346226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.346186 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqwq\" (UniqueName: \"kubernetes.io/projected/83540a58-81a7-41de-b896-cb38b0b0e3ce-kube-api-access-lbqwq\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.346474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.346233 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.346474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.346272 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.346985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.346961 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.348823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.348799 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.358491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.358461 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqwq\" (UniqueName: \"kubernetes.io/projected/83540a58-81a7-41de-b896-cb38b0b0e3ce-kube-api-access-lbqwq\") pod \"maas-api-revoke-sub-c1e271a9-2vbhw\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.517691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.517608 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:42.793042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:42.793012 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw"] Nov 04 01:35:42.794434 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:35:42.794408 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83540a58_81a7_41de_b896_cb38b0b0e3ce.slice/crio-e66caf20534e02dbcd82cb1f4c436ed29e32ae64173cb8cb99ec6bf52a36050c WatchSource:0}: Error finding container e66caf20534e02dbcd82cb1f4c436ed29e32ae64173cb8cb99ec6bf52a36050c: Status 404 returned error can't find the container with id e66caf20534e02dbcd82cb1f4c436ed29e32ae64173cb8cb99ec6bf52a36050c Nov 04 01:35:43.521985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:43.521888 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" event={"ID":"83540a58-81a7-41de-b896-cb38b0b0e3ce","Type":"ContainerStarted","Data":"4b4347a0b5f5fed55eb9578afc5bc7a0d12a28616664f3e85ea9c4d397857b3a"} Nov 04 01:35:43.521985 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:43.521956 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" event={"ID":"83540a58-81a7-41de-b896-cb38b0b0e3ce","Type":"ContainerStarted","Data":"e66caf20534e02dbcd82cb1f4c436ed29e32ae64173cb8cb99ec6bf52a36050c"} Nov 04 01:35:43.543145 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:43.543092 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" podStartSLOduration=1.5430796070000001 podStartE2EDuration="1.543079607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:43.539741081 +0000 UTC m=+1975.562673115" watchObservedRunningTime="2026-11-04 01:35:43.543079607 +0000 UTC m=+1975.566011614" Nov 04 01:35:44.529297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:44.529265 4312 generic.go:356] "Generic (PLEG): container finished" podID="83540a58-81a7-41de-b896-cb38b0b0e3ce" containerID="4b4347a0b5f5fed55eb9578afc5bc7a0d12a28616664f3e85ea9c4d397857b3a" exitCode=0 Nov 04 01:35:44.529710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:44.529325 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" event={"ID":"83540a58-81a7-41de-b896-cb38b0b0e3ce","Type":"ContainerDied","Data":"4b4347a0b5f5fed55eb9578afc5bc7a0d12a28616664f3e85ea9c4d397857b3a"} Nov 04 01:35:45.666461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.666436 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:45.781718 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.781686 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-openshift-ca\") pod \"83540a58-81a7-41de-b896-cb38b0b0e3ce\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " Nov 04 01:35:45.781886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.781773 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-xks-ca\") pod \"83540a58-81a7-41de-b896-cb38b0b0e3ce\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " Nov 04 01:35:45.781886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.781815 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83540a58-81a7-41de-b896-cb38b0b0e3ce-kube-api-access-lbqwq\" (UniqueName: \"kubernetes.io/projected/83540a58-81a7-41de-b896-cb38b0b0e3ce-kube-api-access-lbqwq\") pod \"83540a58-81a7-41de-b896-cb38b0b0e3ce\" (UID: \"83540a58-81a7-41de-b896-cb38b0b0e3ce\") " Nov 04 01:35:45.782097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.782069 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-openshift-ca" pod "83540a58-81a7-41de-b896-cb38b0b0e3ce" (UID: "83540a58-81a7-41de-b896-cb38b0b0e3ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:45.783864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.783846 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83540a58-81a7-41de-b896-cb38b0b0e3ce-kube-api-access-lbqwq" pod "83540a58-81a7-41de-b896-cb38b0b0e3ce" (UID: "83540a58-81a7-41de-b896-cb38b0b0e3ce"). InnerVolumeSpecName "kube-api-access-lbqwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:45.783981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.783960 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-xks-ca" pod "83540a58-81a7-41de-b896-cb38b0b0e3ce" (UID: "83540a58-81a7-41de-b896-cb38b0b0e3ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:45.883110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.883083 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:45.883110 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.883106 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83540a58-81a7-41de-b896-cb38b0b0e3ce-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:45.883274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:45.883118 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqwq\" (UniqueName: \"kubernetes.io/projected/83540a58-81a7-41de-b896-cb38b0b0e3ce-kube-api-access-lbqwq\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:46.539900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:46.539871 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" Nov 04 01:35:46.539900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:46.539887 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw" event={"ID":"83540a58-81a7-41de-b896-cb38b0b0e3ce","Type":"ContainerDied","Data":"e66caf20534e02dbcd82cb1f4c436ed29e32ae64173cb8cb99ec6bf52a36050c"} Nov 04 01:35:46.540285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:46.539930 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e66caf20534e02dbcd82cb1f4c436ed29e32ae64173cb8cb99ec6bf52a36050c" Nov 04 01:35:50.528707 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:50.528678 4312 scope.go:122] "RemoveContainer" containerID="b4cbe68599caa72570d267460eea088bd8eef2b2523592a4017e7e974f6e3fe8" Nov 04 01:35:58.905812 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.905776 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj"] Nov 04 01:35:58.906311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.906201 4312 cpu_manager.go:414] "containerMap: removing container" podUID="83540a58-81a7-41de-b896-cb38b0b0e3ce" containerName="revoke-keys" Nov 04 01:35:58.906311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.906214 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83540a58-81a7-41de-b896-cb38b0b0e3ce" containerName="revoke-keys" Nov 04 01:35:58.906311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.906308 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="83540a58-81a7-41de-b896-cb38b0b0e3ce" containerName="revoke-keys" Nov 04 01:35:58.911012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.910985 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:58.916325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.916296 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:58.916624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.916607 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:35:58.916768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.916687 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:58.918411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.918390 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj"] Nov 04 01:35:58.920453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:58.920428 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:59.062121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.062084 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvl7q\" (UniqueName: \"kubernetes.io/projected/ded242f2-0636-4365-9431-1376f4ad6b37-kube-api-access-jvl7q\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.062288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.062132 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.062357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.062285 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.163017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.162933 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvl7q\" (UniqueName: \"kubernetes.io/projected/ded242f2-0636-4365-9431-1376f4ad6b37-kube-api-access-jvl7q\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.163017 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.162989 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.163251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.163096 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.163685 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.163661 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.165396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.165375 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.175635 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.175604 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvl7q\" (UniqueName: \"kubernetes.io/projected/ded242f2-0636-4365-9431-1376f4ad6b37-kube-api-access-jvl7q\") pod \"maas-api-revoke-sub-2e1f5b5f-c57jj\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.226902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.226871 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:35:59.456760 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:35:59.456670 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:35:59.502029 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.502006 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj"] Nov 04 01:35:59.503850 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:35:59.503826 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded242f2_0636_4365_9431_1376f4ad6b37.slice/crio-c791e5ced998f064635831613aaeae0773ce64e4c717ed3765cbfc3ca1ae56b7 WatchSource:0}: Error finding container c791e5ced998f064635831613aaeae0773ce64e4c717ed3765cbfc3ca1ae56b7: Status 404 returned error can't find the container with id c791e5ced998f064635831613aaeae0773ce64e4c717ed3765cbfc3ca1ae56b7 Nov 04 01:35:59.631197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:35:59.631162 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" event={"ID":"ded242f2-0636-4365-9431-1376f4ad6b37","Type":"ContainerStarted","Data":"c791e5ced998f064635831613aaeae0773ce64e4c717ed3765cbfc3ca1ae56b7"} Nov 04 01:36:00.637214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:00.637180 4312 generic.go:356] "Generic (PLEG): container finished" podID="ded242f2-0636-4365-9431-1376f4ad6b37" containerID="d0a41812665904138afc968293ebbd6b074e14cb4cf76d3dfa7987bb2730fa9e" exitCode=0 Nov 04 01:36:00.637639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:00.637231 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" event={"ID":"ded242f2-0636-4365-9431-1376f4ad6b37","Type":"ContainerDied","Data":"d0a41812665904138afc968293ebbd6b074e14cb4cf76d3dfa7987bb2730fa9e"} Nov 04 01:36:01.772756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.772728 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:36:01.890602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.890561 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded242f2-0636-4365-9431-1376f4ad6b37-kube-api-access-jvl7q\" (UniqueName: \"kubernetes.io/projected/ded242f2-0636-4365-9431-1376f4ad6b37-kube-api-access-jvl7q\") pod \"ded242f2-0636-4365-9431-1376f4ad6b37\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " Nov 04 01:36:01.890814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.890645 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-openshift-ca\") pod \"ded242f2-0636-4365-9431-1376f4ad6b37\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " Nov 04 01:36:01.890814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.890674 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-xks-ca\") pod \"ded242f2-0636-4365-9431-1376f4ad6b37\" (UID: \"ded242f2-0636-4365-9431-1376f4ad6b37\") " Nov 04 01:36:01.891003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.890978 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-openshift-ca" pod "ded242f2-0636-4365-9431-1376f4ad6b37" (UID: "ded242f2-0636-4365-9431-1376f4ad6b37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:01.893011 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.892978 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded242f2-0636-4365-9431-1376f4ad6b37-kube-api-access-jvl7q" pod "ded242f2-0636-4365-9431-1376f4ad6b37" (UID: "ded242f2-0636-4365-9431-1376f4ad6b37"). InnerVolumeSpecName "kube-api-access-jvl7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:01.893173 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.893057 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-xks-ca" pod "ded242f2-0636-4365-9431-1376f4ad6b37" (UID: "ded242f2-0636-4365-9431-1376f4ad6b37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:01.991634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.991595 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.991634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.991630 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded242f2-0636-4365-9431-1376f4ad6b37-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:01.991634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:01.991641 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvl7q\" (UniqueName: \"kubernetes.io/projected/ded242f2-0636-4365-9431-1376f4ad6b37-kube-api-access-jvl7q\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:02.649614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:02.649582 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" Nov 04 01:36:02.649614 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:02.649591 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj" event={"ID":"ded242f2-0636-4365-9431-1376f4ad6b37","Type":"ContainerDied","Data":"c791e5ced998f064635831613aaeae0773ce64e4c717ed3765cbfc3ca1ae56b7"} Nov 04 01:36:02.649892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:02.649643 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c791e5ced998f064635831613aaeae0773ce64e4c717ed3765cbfc3ca1ae56b7" Nov 04 01:36:09.806120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.806085 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv"] Nov 04 01:36:09.806713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.806690 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ded242f2-0636-4365-9431-1376f4ad6b37" containerName="revoke-keys" Nov 04 01:36:09.806759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.806717 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded242f2-0636-4365-9431-1376f4ad6b37" containerName="revoke-keys" Nov 04 01:36:09.806903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.806888 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded242f2-0636-4365-9431-1376f4ad6b37" containerName="revoke-keys" Nov 04 01:36:09.810701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.810684 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.816053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.816030 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:36:09.817556 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.817492 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:36:09.819262 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.819240 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:36:09.819386 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.819284 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:36:09.821948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.821925 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv"] Nov 04 01:36:09.862008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.861974 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kb7s\" (UniqueName: \"kubernetes.io/projected/16d53e28-5a20-486c-9692-6437be49d41c-kube-api-access-6kb7s\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.862169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.862094 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.862169 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.862150 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.963068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.963027 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kb7s\" (UniqueName: \"kubernetes.io/projected/16d53e28-5a20-486c-9692-6437be49d41c-kube-api-access-6kb7s\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.963268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.963142 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.963268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.963188 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.963842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.963820 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.965541 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.965519 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:09.982813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:09.982789 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kb7s\" (UniqueName: \"kubernetes.io/projected/16d53e28-5a20-486c-9692-6437be49d41c-kube-api-access-6kb7s\") pod \"maas-api-revoke-sub-ec843018-8jkxv\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:10.131638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:10.131555 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:10.421879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:10.421849 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv"] Nov 04 01:36:10.423162 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:36:10.423137 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d53e28_5a20_486c_9692_6437be49d41c.slice/crio-872ef733a5a41d3c76124c133d08486ce2e2d4fa2811e4625d5df97205c20ae0 WatchSource:0}: Error finding container 872ef733a5a41d3c76124c133d08486ce2e2d4fa2811e4625d5df97205c20ae0: Status 404 returned error can't find the container with id 872ef733a5a41d3c76124c133d08486ce2e2d4fa2811e4625d5df97205c20ae0 Nov 04 01:36:10.688971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:10.688903 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" event={"ID":"16d53e28-5a20-486c-9692-6437be49d41c","Type":"ContainerStarted","Data":"872ef733a5a41d3c76124c133d08486ce2e2d4fa2811e4625d5df97205c20ae0"} Nov 04 01:36:11.695770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:11.695731 4312 generic.go:356] "Generic (PLEG): container finished" podID="16d53e28-5a20-486c-9692-6437be49d41c" containerID="e29582694b1aa07ed77d39a2ccbb11d04069bbaadcb738bba381cd5875f526e3" exitCode=0 Nov 04 01:36:11.696154 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:11.695786 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" event={"ID":"16d53e28-5a20-486c-9692-6437be49d41c","Type":"ContainerDied","Data":"e29582694b1aa07ed77d39a2ccbb11d04069bbaadcb738bba381cd5875f526e3"} Nov 04 01:36:12.898101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:12.898072 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:12.991083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:12.990983 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d53e28-5a20-486c-9692-6437be49d41c-kube-api-access-6kb7s\" (UniqueName: \"kubernetes.io/projected/16d53e28-5a20-486c-9692-6437be49d41c-kube-api-access-6kb7s\") pod \"16d53e28-5a20-486c-9692-6437be49d41c\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " Nov 04 01:36:12.991083 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:12.991051 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-xks-ca\") pod \"16d53e28-5a20-486c-9692-6437be49d41c\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " Nov 04 01:36:12.991320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:12.991114 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-openshift-ca\") pod \"16d53e28-5a20-486c-9692-6437be49d41c\" (UID: \"16d53e28-5a20-486c-9692-6437be49d41c\") " Nov 04 01:36:12.991846 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:12.991815 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-openshift-ca" pod "16d53e28-5a20-486c-9692-6437be49d41c" (UID: "16d53e28-5a20-486c-9692-6437be49d41c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:12.993822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:12.993791 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-xks-ca" pod "16d53e28-5a20-486c-9692-6437be49d41c" (UID: "16d53e28-5a20-486c-9692-6437be49d41c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:12.993958 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:12.993901 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d53e28-5a20-486c-9692-6437be49d41c-kube-api-access-6kb7s" pod "16d53e28-5a20-486c-9692-6437be49d41c" (UID: "16d53e28-5a20-486c-9692-6437be49d41c"). InnerVolumeSpecName "kube-api-access-6kb7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:13.093422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:13.093384 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kb7s\" (UniqueName: \"kubernetes.io/projected/16d53e28-5a20-486c-9692-6437be49d41c-kube-api-access-6kb7s\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:13.093664 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:13.093644 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:13.093805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:13.093789 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d53e28-5a20-486c-9692-6437be49d41c-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:13.724223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:13.724185 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" event={"ID":"16d53e28-5a20-486c-9692-6437be49d41c","Type":"ContainerDied","Data":"872ef733a5a41d3c76124c133d08486ce2e2d4fa2811e4625d5df97205c20ae0"} Nov 04 01:36:13.724534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:13.724513 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="872ef733a5a41d3c76124c133d08486ce2e2d4fa2811e4625d5df97205c20ae0" Nov 04 01:36:13.724727 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:13.724711 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv" Nov 04 01:36:25.528445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:25.528398 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:36:25.529209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:25.529154 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:36:25.906413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:25.905429 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:36:25.906413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:25.905507 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-ng8g5" podUID="a085702c-fd6e-46d0-b282-1f96cf558e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:36:29.456786 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:36:29.456706 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:36:33.666855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.666819 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg"] Nov 04 01:36:33.667253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.667239 4312 cpu_manager.go:414] "containerMap: removing container" podUID="16d53e28-5a20-486c-9692-6437be49d41c" containerName="revoke-keys" Nov 04 01:36:33.667302 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.667253 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d53e28-5a20-486c-9692-6437be49d41c" containerName="revoke-keys" Nov 04 01:36:33.667361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.667349 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d53e28-5a20-486c-9692-6437be49d41c" containerName="revoke-keys" Nov 04 01:36:33.670820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.670793 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.677193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.676934 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:36:33.677193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.676979 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:36:33.677193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.677051 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:36:33.677193 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.677150 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:36:33.679167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.678974 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg"] Nov 04 01:36:33.832217 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.832181 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.832436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.832242 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z799m\" (UniqueName: \"kubernetes.io/projected/cd272834-20cc-406c-8644-4af71337fc2c-kube-api-access-z799m\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.832494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.832434 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.933673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.933587 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z799m\" (UniqueName: \"kubernetes.io/projected/cd272834-20cc-406c-8644-4af71337fc2c-kube-api-access-z799m\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.933865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.933674 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.933865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.933720 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.934327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.934294 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.936043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.936021 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.946391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.946369 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z799m\" (UniqueName: \"kubernetes.io/projected/cd272834-20cc-406c-8644-4af71337fc2c-kube-api-access-z799m\") pod \"maas-api-revoke-sub-046e1343-xvccg\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:33.986282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:33.986261 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:34.261648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:34.261626 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg"] Nov 04 01:36:34.263404 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:36:34.263373 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd272834_20cc_406c_8644_4af71337fc2c.slice/crio-6fa0fe83318b0e083aefc219c852dd65041302c6cf7b50fe462a139ccfc3adb0 WatchSource:0}: Error finding container 6fa0fe83318b0e083aefc219c852dd65041302c6cf7b50fe462a139ccfc3adb0: Status 404 returned error can't find the container with id 6fa0fe83318b0e083aefc219c852dd65041302c6cf7b50fe462a139ccfc3adb0 Nov 04 01:36:34.862976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:34.862938 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" event={"ID":"cd272834-20cc-406c-8644-4af71337fc2c","Type":"ContainerStarted","Data":"a46bbccac35b8fee833d832807fdd666e8d24ab69d1c5ddd32df481991f8f389"} Nov 04 01:36:34.863398 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:34.862993 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" event={"ID":"cd272834-20cc-406c-8644-4af71337fc2c","Type":"ContainerStarted","Data":"6fa0fe83318b0e083aefc219c852dd65041302c6cf7b50fe462a139ccfc3adb0"} Nov 04 01:36:34.881549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:34.881497 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" podStartSLOduration=1.881483919 podStartE2EDuration="1.881483919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:34.881013131 +0000 UTC m=+2026.903945139" watchObservedRunningTime="2026-11-04 01:36:34.881483919 +0000 UTC m=+2026.904415926" Nov 04 01:36:35.869266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:35.869226 4312 generic.go:356] "Generic (PLEG): container finished" podID="cd272834-20cc-406c-8644-4af71337fc2c" containerID="a46bbccac35b8fee833d832807fdd666e8d24ab69d1c5ddd32df481991f8f389" exitCode=0 Nov 04 01:36:35.869704 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:35.869308 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" event={"ID":"cd272834-20cc-406c-8644-4af71337fc2c","Type":"ContainerDied","Data":"a46bbccac35b8fee833d832807fdd666e8d24ab69d1c5ddd32df481991f8f389"} Nov 04 01:36:37.002950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.002925 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:37.164301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.164216 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-openshift-ca\") pod \"cd272834-20cc-406c-8644-4af71337fc2c\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " Nov 04 01:36:37.164301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.164257 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-xks-ca\") pod \"cd272834-20cc-406c-8644-4af71337fc2c\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " Nov 04 01:36:37.164301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.164304 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd272834-20cc-406c-8644-4af71337fc2c-kube-api-access-z799m\" (UniqueName: \"kubernetes.io/projected/cd272834-20cc-406c-8644-4af71337fc2c-kube-api-access-z799m\") pod \"cd272834-20cc-406c-8644-4af71337fc2c\" (UID: \"cd272834-20cc-406c-8644-4af71337fc2c\") " Nov 04 01:36:37.164700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.164672 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-openshift-ca" pod "cd272834-20cc-406c-8644-4af71337fc2c" (UID: "cd272834-20cc-406c-8644-4af71337fc2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:37.166475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.166453 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd272834-20cc-406c-8644-4af71337fc2c-kube-api-access-z799m" pod "cd272834-20cc-406c-8644-4af71337fc2c" (UID: "cd272834-20cc-406c-8644-4af71337fc2c"). InnerVolumeSpecName "kube-api-access-z799m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:37.166538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.166473 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-xks-ca" pod "cd272834-20cc-406c-8644-4af71337fc2c" (UID: "cd272834-20cc-406c-8644-4af71337fc2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:37.265354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.265310 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:37.265533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.265366 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd272834-20cc-406c-8644-4af71337fc2c-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:37.265533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.265377 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z799m\" (UniqueName: \"kubernetes.io/projected/cd272834-20cc-406c-8644-4af71337fc2c-kube-api-access-z799m\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:37.880054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.880026 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" Nov 04 01:36:37.880243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.880057 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg" event={"ID":"cd272834-20cc-406c-8644-4af71337fc2c","Type":"ContainerDied","Data":"6fa0fe83318b0e083aefc219c852dd65041302c6cf7b50fe462a139ccfc3adb0"} Nov 04 01:36:37.880243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:37.880110 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fa0fe83318b0e083aefc219c852dd65041302c6cf7b50fe462a139ccfc3adb0" Nov 04 01:36:43.883709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.883675 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc"] Nov 04 01:36:43.884270 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.884104 4312 cpu_manager.go:414] "containerMap: removing container" podUID="cd272834-20cc-406c-8644-4af71337fc2c" containerName="revoke-keys" Nov 04 01:36:43.884270 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.884114 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd272834-20cc-406c-8644-4af71337fc2c" containerName="revoke-keys" Nov 04 01:36:43.884270 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.884186 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd272834-20cc-406c-8644-4af71337fc2c" containerName="revoke-keys" Nov 04 01:36:43.887408 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.887384 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:43.893200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.893140 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:36:43.893924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.893897 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:36:43.899372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.895934 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:36:43.899372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.896014 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:36:43.899372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:43.898933 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc"] Nov 04 01:36:44.035317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.035280 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.035517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.035442 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsgd\" (UniqueName: \"kubernetes.io/projected/16c4dd25-38bf-4580-b4e4-877d023d4044-kube-api-access-5gsgd\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.035517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.035477 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.136782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.136691 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsgd\" (UniqueName: \"kubernetes.io/projected/16c4dd25-38bf-4580-b4e4-877d023d4044-kube-api-access-5gsgd\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.136782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.136741 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.136987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.136792 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.137551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.137526 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.139181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.139150 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.149499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.149475 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsgd\" (UniqueName: \"kubernetes.io/projected/16c4dd25-38bf-4580-b4e4-877d023d4044-kube-api-access-5gsgd\") pod \"maas-api-revoke-sub-d3fba056-fdwbc\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.204097 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.204063 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:44.483430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.483403 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc"] Nov 04 01:36:44.485545 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:36:44.485516 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c4dd25_38bf_4580_b4e4_877d023d4044.slice/crio-171b10c6ebf8df5bbd40db0bc0ab9ad6e922fa5b57c3cdcc4a4e7d550a802df6 WatchSource:0}: Error finding container 171b10c6ebf8df5bbd40db0bc0ab9ad6e922fa5b57c3cdcc4a4e7d550a802df6: Status 404 returned error can't find the container with id 171b10c6ebf8df5bbd40db0bc0ab9ad6e922fa5b57c3cdcc4a4e7d550a802df6 Nov 04 01:36:44.920575 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.920536 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" event={"ID":"16c4dd25-38bf-4580-b4e4-877d023d4044","Type":"ContainerStarted","Data":"014ae7e1328aaa60bdadb622f85de9b8941268f4e5fa2ecd482731be032e48fa"} Nov 04 01:36:44.920997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.920592 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" event={"ID":"16c4dd25-38bf-4580-b4e4-877d023d4044","Type":"ContainerStarted","Data":"171b10c6ebf8df5bbd40db0bc0ab9ad6e922fa5b57c3cdcc4a4e7d550a802df6"} Nov 04 01:36:44.941606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:44.941530 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" podStartSLOduration=1.941509286 podStartE2EDuration="1.941509286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:44.940711237 +0000 UTC m=+2036.963643266" watchObservedRunningTime="2026-11-04 01:36:44.941509286 +0000 UTC m=+2036.964441294" Nov 04 01:36:45.940456 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:45.940417 4312 generic.go:356] "Generic (PLEG): container finished" podID="16c4dd25-38bf-4580-b4e4-877d023d4044" containerID="014ae7e1328aaa60bdadb622f85de9b8941268f4e5fa2ecd482731be032e48fa" exitCode=0 Nov 04 01:36:45.941092 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:45.940526 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" event={"ID":"16c4dd25-38bf-4580-b4e4-877d023d4044","Type":"ContainerDied","Data":"014ae7e1328aaa60bdadb622f85de9b8941268f4e5fa2ecd482731be032e48fa"} Nov 04 01:36:47.529530 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.529467 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:36:47.530203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.530148 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:36:47.815668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.815298 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:47.911360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.908715 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-xks-ca\") pod \"16c4dd25-38bf-4580-b4e4-877d023d4044\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " Nov 04 01:36:47.911360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.908801 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-openshift-ca\") pod \"16c4dd25-38bf-4580-b4e4-877d023d4044\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " Nov 04 01:36:47.911360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.908875 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c4dd25-38bf-4580-b4e4-877d023d4044-kube-api-access-5gsgd\" (UniqueName: \"kubernetes.io/projected/16c4dd25-38bf-4580-b4e4-877d023d4044-kube-api-access-5gsgd\") pod \"16c4dd25-38bf-4580-b4e4-877d023d4044\" (UID: \"16c4dd25-38bf-4580-b4e4-877d023d4044\") " Nov 04 01:36:47.916354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.915212 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-openshift-ca" pod "16c4dd25-38bf-4580-b4e4-877d023d4044" (UID: "16c4dd25-38bf-4580-b4e4-877d023d4044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:47.929906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.929840 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-xks-ca" pod "16c4dd25-38bf-4580-b4e4-877d023d4044" (UID: "16c4dd25-38bf-4580-b4e4-877d023d4044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:47.935947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:47.935888 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c4dd25-38bf-4580-b4e4-877d023d4044-kube-api-access-5gsgd" pod "16c4dd25-38bf-4580-b4e4-877d023d4044" (UID: "16c4dd25-38bf-4580-b4e4-877d023d4044"). InnerVolumeSpecName "kube-api-access-5gsgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:48.003195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:48.003123 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" event={"ID":"16c4dd25-38bf-4580-b4e4-877d023d4044","Type":"ContainerDied","Data":"171b10c6ebf8df5bbd40db0bc0ab9ad6e922fa5b57c3cdcc4a4e7d550a802df6"} Nov 04 01:36:48.007366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:48.003585 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="171b10c6ebf8df5bbd40db0bc0ab9ad6e922fa5b57c3cdcc4a4e7d550a802df6" Nov 04 01:36:48.007366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:48.003698 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc" Nov 04 01:36:48.014034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:48.013978 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:48.014034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:48.014009 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c4dd25-38bf-4580-b4e4-877d023d4044-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:48.014034 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:36:48.014025 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsgd\" (UniqueName: \"kubernetes.io/projected/16c4dd25-38bf-4580-b4e4-877d023d4044-kube-api-access-5gsgd\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:59.545714 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:36:59.545668 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:37:04.490300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.490268 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2"] Nov 04 01:37:04.490751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.490730 4312 cpu_manager.go:414] "containerMap: removing container" podUID="16c4dd25-38bf-4580-b4e4-877d023d4044" containerName="revoke-keys" Nov 04 01:37:04.490814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.490753 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c4dd25-38bf-4580-b4e4-877d023d4044" containerName="revoke-keys" Nov 04 01:37:04.490876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.490864 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c4dd25-38bf-4580-b4e4-877d023d4044" containerName="revoke-keys" Nov 04 01:37:04.494323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.494306 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.500460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.500438 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:37:04.502378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.502357 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:04.502492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.502457 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:37:04.502540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.502502 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:37:04.504829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.504807 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2"] Nov 04 01:37:04.530298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.530269 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6ng\" (UniqueName: \"kubernetes.io/projected/ff67d663-0c56-4b03-a1a3-db946bab457a-kube-api-access-ht6ng\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.530470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.530454 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.530518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.530479 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.630895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.630862 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.630895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.630895 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.631176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.630926 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6ng\" (UniqueName: \"kubernetes.io/projected/ff67d663-0c56-4b03-a1a3-db946bab457a-kube-api-access-ht6ng\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.631644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.631611 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.633313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.633290 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.642858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.642833 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6ng\" (UniqueName: \"kubernetes.io/projected/ff67d663-0c56-4b03-a1a3-db946bab457a-kube-api-access-ht6ng\") pod \"maas-api-revoke-sub-ae594979-jxmr2\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:04.810240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:04.810210 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:05.086905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:05.086868 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2"] Nov 04 01:37:05.088822 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:37:05.088791 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff67d663_0c56_4b03_a1a3_db946bab457a.slice/crio-e49946bc64b54c6eed5addf3d8318dfbb527f5ec867f5c5aadf9790e387dc681 WatchSource:0}: Error finding container e49946bc64b54c6eed5addf3d8318dfbb527f5ec867f5c5aadf9790e387dc681: Status 404 returned error can't find the container with id e49946bc64b54c6eed5addf3d8318dfbb527f5ec867f5c5aadf9790e387dc681 Nov 04 01:37:05.128473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:05.128447 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" event={"ID":"ff67d663-0c56-4b03-a1a3-db946bab457a","Type":"ContainerStarted","Data":"e49946bc64b54c6eed5addf3d8318dfbb527f5ec867f5c5aadf9790e387dc681"} Nov 04 01:37:06.134342 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:06.134312 4312 generic.go:356] "Generic (PLEG): container finished" podID="ff67d663-0c56-4b03-a1a3-db946bab457a" containerID="2f4d2caceb1f89ccf774dfbc29f434f88a393b139fef08f5a07adc4236aba9a9" exitCode=0 Nov 04 01:37:06.134743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:06.134401 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" event={"ID":"ff67d663-0c56-4b03-a1a3-db946bab457a","Type":"ContainerDied","Data":"2f4d2caceb1f89ccf774dfbc29f434f88a393b139fef08f5a07adc4236aba9a9"} Nov 04 01:37:07.273493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.273470 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:07.352134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.352107 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff67d663-0c56-4b03-a1a3-db946bab457a-kube-api-access-ht6ng\" (UniqueName: \"kubernetes.io/projected/ff67d663-0c56-4b03-a1a3-db946bab457a-kube-api-access-ht6ng\") pod \"ff67d663-0c56-4b03-a1a3-db946bab457a\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " Nov 04 01:37:07.352288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.352170 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-xks-ca\") pod \"ff67d663-0c56-4b03-a1a3-db946bab457a\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " Nov 04 01:37:07.352288 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.352200 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-openshift-ca\") pod \"ff67d663-0c56-4b03-a1a3-db946bab457a\" (UID: \"ff67d663-0c56-4b03-a1a3-db946bab457a\") " Nov 04 01:37:07.352638 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.352616 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-openshift-ca" pod "ff67d663-0c56-4b03-a1a3-db946bab457a" (UID: "ff67d663-0c56-4b03-a1a3-db946bab457a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:07.354118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.354100 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-xks-ca" pod "ff67d663-0c56-4b03-a1a3-db946bab457a" (UID: "ff67d663-0c56-4b03-a1a3-db946bab457a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:07.354226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.354206 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff67d663-0c56-4b03-a1a3-db946bab457a-kube-api-access-ht6ng" pod "ff67d663-0c56-4b03-a1a3-db946bab457a" (UID: "ff67d663-0c56-4b03-a1a3-db946bab457a"). InnerVolumeSpecName "kube-api-access-ht6ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:07.453383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.453307 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:07.453383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.453350 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht6ng\" (UniqueName: \"kubernetes.io/projected/ff67d663-0c56-4b03-a1a3-db946bab457a-kube-api-access-ht6ng\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:07.453383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:07.453366 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff67d663-0c56-4b03-a1a3-db946bab457a-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:08.146869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:08.146847 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" Nov 04 01:37:08.147031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:08.146838 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2" event={"ID":"ff67d663-0c56-4b03-a1a3-db946bab457a","Type":"ContainerDied","Data":"e49946bc64b54c6eed5addf3d8318dfbb527f5ec867f5c5aadf9790e387dc681"} Nov 04 01:37:08.147031 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:08.146980 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49946bc64b54c6eed5addf3d8318dfbb527f5ec867f5c5aadf9790e387dc681" Nov 04 01:37:21.242593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.242560 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb"] Nov 04 01:37:21.243033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.243018 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ff67d663-0c56-4b03-a1a3-db946bab457a" containerName="revoke-keys" Nov 04 01:37:21.243079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.243033 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff67d663-0c56-4b03-a1a3-db946bab457a" containerName="revoke-keys" Nov 04 01:37:21.243124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.243106 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff67d663-0c56-4b03-a1a3-db946bab457a" containerName="revoke-keys" Nov 04 01:37:21.246598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.246577 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.253163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.252327 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:21.253163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.252918 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:37:21.254362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.254312 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:37:21.254362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.254323 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:37:21.257294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.257274 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb"] Nov 04 01:37:21.314225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.314190 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.314451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.314261 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwlk\" (UniqueName: \"kubernetes.io/projected/7532eea5-d5f2-45e1-9d75-cb09717aeba5-kube-api-access-tcwlk\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.314451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.314408 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.415139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.415105 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.415380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.415174 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwlk\" (UniqueName: \"kubernetes.io/projected/7532eea5-d5f2-45e1-9d75-cb09717aeba5-kube-api-access-tcwlk\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.415380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.415265 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.415974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.415946 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.417608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.417576 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.427909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.427888 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwlk\" (UniqueName: \"kubernetes.io/projected/7532eea5-d5f2-45e1-9d75-cb09717aeba5-kube-api-access-tcwlk\") pod \"maas-api-revoke-sub-a4aef7d7-t6rgb\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.561433 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.561405 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:21.838178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:21.838144 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb"] Nov 04 01:37:21.839687 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:37:21.839657 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7532eea5_d5f2_45e1_9d75_cb09717aeba5.slice/crio-39dcca581bd4e59362a95b719b539a5098a28e4e4e02dedb59584371689f3536 WatchSource:0}: Error finding container 39dcca581bd4e59362a95b719b539a5098a28e4e4e02dedb59584371689f3536: Status 404 returned error can't find the container with id 39dcca581bd4e59362a95b719b539a5098a28e4e4e02dedb59584371689f3536 Nov 04 01:37:22.247322 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:22.247290 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" event={"ID":"7532eea5-d5f2-45e1-9d75-cb09717aeba5","Type":"ContainerStarted","Data":"23b3e77c165609b62d0bafad210d9f3262fb62d264e79f73657972abd80d85e1"} Nov 04 01:37:22.247770 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:22.247380 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" event={"ID":"7532eea5-d5f2-45e1-9d75-cb09717aeba5","Type":"ContainerStarted","Data":"39dcca581bd4e59362a95b719b539a5098a28e4e4e02dedb59584371689f3536"} Nov 04 01:37:22.266066 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:22.266008 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" podStartSLOduration=1.265992332 podStartE2EDuration="1.265992332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:22.264973267 +0000 UTC m=+2074.287905275" watchObservedRunningTime="2026-11-04 01:37:22.265992332 +0000 UTC m=+2074.288924339" Nov 04 01:37:23.253877 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:23.253846 4312 generic.go:356] "Generic (PLEG): container finished" podID="7532eea5-d5f2-45e1-9d75-cb09717aeba5" containerID="23b3e77c165609b62d0bafad210d9f3262fb62d264e79f73657972abd80d85e1" exitCode=0 Nov 04 01:37:23.254279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:23.253909 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" event={"ID":"7532eea5-d5f2-45e1-9d75-cb09717aeba5","Type":"ContainerDied","Data":"23b3e77c165609b62d0bafad210d9f3262fb62d264e79f73657972abd80d85e1"} Nov 04 01:37:24.401415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.401388 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:24.446749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.446719 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-openshift-ca\") pod \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " Nov 04 01:37:24.446898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.446792 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7532eea5-d5f2-45e1-9d75-cb09717aeba5-kube-api-access-tcwlk\" (UniqueName: \"kubernetes.io/projected/7532eea5-d5f2-45e1-9d75-cb09717aeba5-kube-api-access-tcwlk\") pod \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " Nov 04 01:37:24.446898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.446816 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-xks-ca\") pod \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\" (UID: \"7532eea5-d5f2-45e1-9d75-cb09717aeba5\") " Nov 04 01:37:24.447139 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.447111 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-openshift-ca" pod "7532eea5-d5f2-45e1-9d75-cb09717aeba5" (UID: "7532eea5-d5f2-45e1-9d75-cb09717aeba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:24.448973 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.448941 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-xks-ca" pod "7532eea5-d5f2-45e1-9d75-cb09717aeba5" (UID: "7532eea5-d5f2-45e1-9d75-cb09717aeba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:24.449100 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.449025 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7532eea5-d5f2-45e1-9d75-cb09717aeba5-kube-api-access-tcwlk" pod "7532eea5-d5f2-45e1-9d75-cb09717aeba5" (UID: "7532eea5-d5f2-45e1-9d75-cb09717aeba5"). InnerVolumeSpecName "kube-api-access-tcwlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:24.548269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.548235 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:24.548269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.548266 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcwlk\" (UniqueName: \"kubernetes.io/projected/7532eea5-d5f2-45e1-9d75-cb09717aeba5-kube-api-access-tcwlk\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:24.548269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:24.548276 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7532eea5-d5f2-45e1-9d75-cb09717aeba5-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:25.265446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:25.265415 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" Nov 04 01:37:25.265624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:25.265450 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb" event={"ID":"7532eea5-d5f2-45e1-9d75-cb09717aeba5","Type":"ContainerDied","Data":"39dcca581bd4e59362a95b719b539a5098a28e4e4e02dedb59584371689f3536"} Nov 04 01:37:25.265624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:25.265500 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39dcca581bd4e59362a95b719b539a5098a28e4e4e02dedb59584371689f3536" Nov 04 01:37:29.457148 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:37:29.457110 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:37:36.762364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.758412 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8"] Nov 04 01:37:36.762364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.759081 4312 cpu_manager.go:414] "containerMap: removing container" podUID="7532eea5-d5f2-45e1-9d75-cb09717aeba5" containerName="revoke-keys" Nov 04 01:37:36.762364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.759101 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7532eea5-d5f2-45e1-9d75-cb09717aeba5" containerName="revoke-keys" Nov 04 01:37:36.762364 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.759262 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="7532eea5-d5f2-45e1-9d75-cb09717aeba5" containerName="revoke-keys" Nov 04 01:37:36.766388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.763200 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8"] Nov 04 01:37:36.766388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.763320 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:36.777359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.772126 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:37:36.777679 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.777652 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:37:36.777855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.777840 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:36.778047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.778031 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:37:36.899472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.899422 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:36.899892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.899851 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:36.900061 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:36.900044 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6cv\" (UniqueName: \"kubernetes.io/projected/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-kube-api-access-vh6cv\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.020178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.020080 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.020569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.020545 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.020750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.020730 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6cv\" (UniqueName: \"kubernetes.io/projected/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-kube-api-access-vh6cv\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.022938 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.022916 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.044101 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.044074 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6cv\" (UniqueName: \"kubernetes.io/projected/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-kube-api-access-vh6cv\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.060889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.060862 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fz7d8\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.150560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.150527 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:37.680460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:37.680238 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8"] Nov 04 01:37:38.351033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:38.350995 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" event={"ID":"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2","Type":"ContainerStarted","Data":"285674212df471aecfb6464fddca9effe87d552573e2cf3b138b143865ac9d3c"} Nov 04 01:37:38.351440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:38.351062 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" event={"ID":"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2","Type":"ContainerStarted","Data":"bbb4245a0954ddb0c8dfb3f1e9722c29576766ab25aa62e2a943d4380a007ddd"} Nov 04 01:37:38.371211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:38.371158 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" podStartSLOduration=2.371144283 podStartE2EDuration="2.371144283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:38.370875113 +0000 UTC m=+2090.393807120" watchObservedRunningTime="2026-11-04 01:37:38.371144283 +0000 UTC m=+2090.394076292" Nov 04 01:37:39.357046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:39.357012 4312 generic.go:356] "Generic (PLEG): container finished" podID="2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" containerID="285674212df471aecfb6464fddca9effe87d552573e2cf3b138b143865ac9d3c" exitCode=0 Nov 04 01:37:39.357478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:39.357055 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" event={"ID":"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2","Type":"ContainerDied","Data":"285674212df471aecfb6464fddca9effe87d552573e2cf3b138b143865ac9d3c"} Nov 04 01:37:40.500864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.500842 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:40.593690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.593658 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-kube-api-access-vh6cv\" (UniqueName: \"kubernetes.io/projected/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-kube-api-access-vh6cv\") pod \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " Nov 04 01:37:40.593844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.593749 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-xks-ca\") pod \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " Nov 04 01:37:40.593844 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.593786 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-openshift-ca\") pod \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\" (UID: \"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2\") " Nov 04 01:37:40.594192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.594168 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-openshift-ca" pod "2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" (UID: "2aaa7aff-0210-4ed9-bb4c-5b796375d1d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:40.595803 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.595774 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-xks-ca" pod "2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" (UID: "2aaa7aff-0210-4ed9-bb4c-5b796375d1d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:40.595926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.595819 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-kube-api-access-vh6cv" pod "2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" (UID: "2aaa7aff-0210-4ed9-bb4c-5b796375d1d2"). InnerVolumeSpecName "kube-api-access-vh6cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:40.695087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.695019 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:40.695087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.695044 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:40.695087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:40.695055 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6cv\" (UniqueName: \"kubernetes.io/projected/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2-kube-api-access-vh6cv\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:41.368418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:41.368392 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" Nov 04 01:37:41.368418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:41.368400 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8" event={"ID":"2aaa7aff-0210-4ed9-bb4c-5b796375d1d2","Type":"ContainerDied","Data":"bbb4245a0954ddb0c8dfb3f1e9722c29576766ab25aa62e2a943d4380a007ddd"} Nov 04 01:37:41.368626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:41.368446 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbb4245a0954ddb0c8dfb3f1e9722c29576766ab25aa62e2a943d4380a007ddd" Nov 04 01:37:51.337747 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.337695 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92"] Nov 04 01:37:51.338959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.338933 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" containerName="revoke-keys" Nov 04 01:37:51.339119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.339102 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" containerName="revoke-keys" Nov 04 01:37:51.339374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.339357 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" containerName="revoke-keys" Nov 04 01:37:51.401357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.391666 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.414358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.405572 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:51.414358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.409635 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:37:51.433355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.416498 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92"] Nov 04 01:37:51.433355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.417377 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:37:51.439785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.439762 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:37:51.463379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.448879 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.463379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.449030 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.463379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.449078 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv4sb\" (UniqueName: \"kubernetes.io/projected/0bf43bbb-c925-42c4-aff8-fa7cc5616087-kube-api-access-jv4sb\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.551706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.551662 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.588703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.552682 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.589513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.588979 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.590297 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.590272 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv4sb\" (UniqueName: \"kubernetes.io/projected/0bf43bbb-c925-42c4-aff8-fa7cc5616087-kube-api-access-jv4sb\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.662361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.652143 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv4sb\" (UniqueName: \"kubernetes.io/projected/0bf43bbb-c925-42c4-aff8-fa7cc5616087-kube-api-access-jv4sb\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.662648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.659731 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-5gw92\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:51.831357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:51.830787 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:52.556357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:52.540411 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92"] Nov 04 01:37:52.722437 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:37:52.714132 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf43bbb_c925_42c4_aff8_fa7cc5616087.slice/crio-a988fc344e4847b1ef3f7ab0e173cdb48a9ad239b353c869eddde7373bd0447e WatchSource:0}: Error finding container a988fc344e4847b1ef3f7ab0e173cdb48a9ad239b353c869eddde7373bd0447e: Status 404 returned error can't find the container with id a988fc344e4847b1ef3f7ab0e173cdb48a9ad239b353c869eddde7373bd0447e Nov 04 01:37:53.647190 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:53.647157 4312 generic.go:356] "Generic (PLEG): container finished" podID="0bf43bbb-c925-42c4-aff8-fa7cc5616087" containerID="fb4b08e52fedb6885050e35aeb7fedef9d2fd37333ba8180a155e74d95a3e59a" exitCode=0 Nov 04 01:37:53.647569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:53.647239 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" event={"ID":"0bf43bbb-c925-42c4-aff8-fa7cc5616087","Type":"ContainerDied","Data":"fb4b08e52fedb6885050e35aeb7fedef9d2fd37333ba8180a155e74d95a3e59a"} Nov 04 01:37:53.647569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:53.647295 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" event={"ID":"0bf43bbb-c925-42c4-aff8-fa7cc5616087","Type":"ContainerStarted","Data":"a988fc344e4847b1ef3f7ab0e173cdb48a9ad239b353c869eddde7373bd0447e"} Nov 04 01:37:54.785322 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.785297 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:54.872428 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.872402 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-openshift-ca\") pod \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " Nov 04 01:37:54.872606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.872519 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf43bbb-c925-42c4-aff8-fa7cc5616087-kube-api-access-jv4sb\" (UniqueName: \"kubernetes.io/projected/0bf43bbb-c925-42c4-aff8-fa7cc5616087-kube-api-access-jv4sb\") pod \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " Nov 04 01:37:54.872606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.872556 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-xks-ca\") pod \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\" (UID: \"0bf43bbb-c925-42c4-aff8-fa7cc5616087\") " Nov 04 01:37:54.872752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.872736 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-openshift-ca" pod "0bf43bbb-c925-42c4-aff8-fa7cc5616087" (UID: "0bf43bbb-c925-42c4-aff8-fa7cc5616087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:54.872827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.872813 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:54.874645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.874614 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf43bbb-c925-42c4-aff8-fa7cc5616087-kube-api-access-jv4sb" pod "0bf43bbb-c925-42c4-aff8-fa7cc5616087" (UID: "0bf43bbb-c925-42c4-aff8-fa7cc5616087"). InnerVolumeSpecName "kube-api-access-jv4sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:54.874757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.874679 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-xks-ca" pod "0bf43bbb-c925-42c4-aff8-fa7cc5616087" (UID: "0bf43bbb-c925-42c4-aff8-fa7cc5616087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:54.974026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.973941 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv4sb\" (UniqueName: \"kubernetes.io/projected/0bf43bbb-c925-42c4-aff8-fa7cc5616087-kube-api-access-jv4sb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:54.974026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:54.973973 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf43bbb-c925-42c4-aff8-fa7cc5616087-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:55.658606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:55.658574 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" Nov 04 01:37:55.658796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:55.658610 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92" event={"ID":"0bf43bbb-c925-42c4-aff8-fa7cc5616087","Type":"ContainerDied","Data":"a988fc344e4847b1ef3f7ab0e173cdb48a9ad239b353c869eddde7373bd0447e"} Nov 04 01:37:55.658796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:37:55.658670 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a988fc344e4847b1ef3f7ab0e173cdb48a9ad239b353c869eddde7373bd0447e" Nov 04 01:37:59.457602 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:37:59.457524 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:38:01.974132 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.974094 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b"] Nov 04 01:38:01.974598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.974583 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0bf43bbb-c925-42c4-aff8-fa7cc5616087" containerName="revoke-keys" Nov 04 01:38:01.974598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.974598 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf43bbb-c925-42c4-aff8-fa7cc5616087" containerName="revoke-keys" Nov 04 01:38:01.974690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.974675 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf43bbb-c925-42c4-aff8-fa7cc5616087" containerName="revoke-keys" Nov 04 01:38:01.978177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.978158 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:01.983914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.983874 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:01.984119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.983945 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:38:01.984119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.984080 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:38:01.984983 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.984962 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:38:01.988135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:01.988115 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b"] Nov 04 01:38:02.042567 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.042531 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.042773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.042579 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.042773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.042665 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmk2\" (UniqueName: \"kubernetes.io/projected/3526d040-0df1-4795-8a36-6aea0141f3ef-kube-api-access-jqmk2\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.143749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.143715 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.143749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.143761 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.143982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.143781 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmk2\" (UniqueName: \"kubernetes.io/projected/3526d040-0df1-4795-8a36-6aea0141f3ef-kube-api-access-jqmk2\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.144434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.144411 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.146175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.146154 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.158004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.157973 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmk2\" (UniqueName: \"kubernetes.io/projected/3526d040-0df1-4795-8a36-6aea0141f3ef-kube-api-access-jqmk2\") pod \"maas-api-revoke-sub-3d4156f1-45g6b\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.295869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.295834 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:02.576266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.576230 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b"] Nov 04 01:38:02.578042 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:38:02.578014 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3526d040_0df1_4795_8a36_6aea0141f3ef.slice/crio-6d4fb91a521af6c4a9e2c49d8347381f5ab406da703d3a6bbfe363b7b7e9e0e6 WatchSource:0}: Error finding container 6d4fb91a521af6c4a9e2c49d8347381f5ab406da703d3a6bbfe363b7b7e9e0e6: Status 404 returned error can't find the container with id 6d4fb91a521af6c4a9e2c49d8347381f5ab406da703d3a6bbfe363b7b7e9e0e6 Nov 04 01:38:02.696107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:02.696072 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" event={"ID":"3526d040-0df1-4795-8a36-6aea0141f3ef","Type":"ContainerStarted","Data":"6d4fb91a521af6c4a9e2c49d8347381f5ab406da703d3a6bbfe363b7b7e9e0e6"} Nov 04 01:38:03.703212 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:03.703129 4312 generic.go:356] "Generic (PLEG): container finished" podID="3526d040-0df1-4795-8a36-6aea0141f3ef" containerID="052a660ba2bd9b6f0ad0cc489adaefafc94958d005c18b072d1a2edc2119c2ea" exitCode=0 Nov 04 01:38:03.703631 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:03.703215 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" event={"ID":"3526d040-0df1-4795-8a36-6aea0141f3ef","Type":"ContainerDied","Data":"052a660ba2bd9b6f0ad0cc489adaefafc94958d005c18b072d1a2edc2119c2ea"} Nov 04 01:38:04.848013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:04.847987 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:04.973389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:04.973276 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3526d040-0df1-4795-8a36-6aea0141f3ef-kube-api-access-jqmk2\" (UniqueName: \"kubernetes.io/projected/3526d040-0df1-4795-8a36-6aea0141f3ef-kube-api-access-jqmk2\") pod \"3526d040-0df1-4795-8a36-6aea0141f3ef\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " Nov 04 01:38:04.973389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:04.973386 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-xks-ca\") pod \"3526d040-0df1-4795-8a36-6aea0141f3ef\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " Nov 04 01:38:04.973646 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:04.973432 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-openshift-ca\") pod \"3526d040-0df1-4795-8a36-6aea0141f3ef\" (UID: \"3526d040-0df1-4795-8a36-6aea0141f3ef\") " Nov 04 01:38:04.973833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:04.973799 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-openshift-ca" pod "3526d040-0df1-4795-8a36-6aea0141f3ef" (UID: "3526d040-0df1-4795-8a36-6aea0141f3ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:04.975560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:04.975538 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-xks-ca" pod "3526d040-0df1-4795-8a36-6aea0141f3ef" (UID: "3526d040-0df1-4795-8a36-6aea0141f3ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:04.975672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:04.975650 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3526d040-0df1-4795-8a36-6aea0141f3ef-kube-api-access-jqmk2" pod "3526d040-0df1-4795-8a36-6aea0141f3ef" (UID: "3526d040-0df1-4795-8a36-6aea0141f3ef"). InnerVolumeSpecName "kube-api-access-jqmk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:05.074888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:05.074851 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:05.074888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:05.074882 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3526d040-0df1-4795-8a36-6aea0141f3ef-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:05.074888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:05.074893 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmk2\" (UniqueName: \"kubernetes.io/projected/3526d040-0df1-4795-8a36-6aea0141f3ef-kube-api-access-jqmk2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:05.714307 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:05.714278 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" Nov 04 01:38:05.714492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:05.714309 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b" event={"ID":"3526d040-0df1-4795-8a36-6aea0141f3ef","Type":"ContainerDied","Data":"6d4fb91a521af6c4a9e2c49d8347381f5ab406da703d3a6bbfe363b7b7e9e0e6"} Nov 04 01:38:05.714492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:05.714371 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d4fb91a521af6c4a9e2c49d8347381f5ab406da703d3a6bbfe363b7b7e9e0e6" Nov 04 01:38:16.851035 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.850994 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b"] Nov 04 01:38:16.854929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.854897 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-45g6b"] Nov 04 01:38:16.876581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.876546 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz"] Nov 04 01:38:16.877000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.876986 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3526d040-0df1-4795-8a36-6aea0141f3ef" containerName="revoke-keys" Nov 04 01:38:16.877051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.877001 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3526d040-0df1-4795-8a36-6aea0141f3ef" containerName="revoke-keys" Nov 04 01:38:16.877100 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.877090 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3526d040-0df1-4795-8a36-6aea0141f3ef" containerName="revoke-keys" Nov 04 01:38:16.880380 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.880361 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:16.886298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.886266 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:38:16.886298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.886267 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:16.888414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.887879 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:38:16.888507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.888444 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:38:16.891905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:16.891878 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz"] Nov 04 01:38:17.033564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.033529 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnbm\" (UniqueName: \"kubernetes.io/projected/ccbd0d38-f9ee-40d5-af7a-0688d085a961-kube-api-access-kcnbm\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.033741 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.033573 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.033741 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.033699 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.135167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.135090 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnbm\" (UniqueName: \"kubernetes.io/projected/ccbd0d38-f9ee-40d5-af7a-0688d085a961-kube-api-access-kcnbm\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.135167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.135127 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.135393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.135177 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.135856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.135833 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.137510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.137490 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.147160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.147137 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnbm\" (UniqueName: \"kubernetes.io/projected/ccbd0d38-f9ee-40d5-af7a-0688d085a961-kube-api-access-kcnbm\") pod \"maas-api-revoke-sub-3d4156f1-hl8vz\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.195472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.195444 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:17.470259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.470233 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz"] Nov 04 01:38:17.472154 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:38:17.472120 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbd0d38_f9ee_40d5_af7a_0688d085a961.slice/crio-87eaf130179eedfa0a71d8392b48e1bf46a1596255499b297f7adff97089bb9c WatchSource:0}: Error finding container 87eaf130179eedfa0a71d8392b48e1bf46a1596255499b297f7adff97089bb9c: Status 404 returned error can't find the container with id 87eaf130179eedfa0a71d8392b48e1bf46a1596255499b297f7adff97089bb9c Nov 04 01:38:17.800107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.800068 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" event={"ID":"ccbd0d38-f9ee-40d5-af7a-0688d085a961","Type":"ContainerStarted","Data":"d2a7d98f458652c248e5f9b1dc8cf57afa5a900bee77d6ad9dca2d5afd2f54f0"} Nov 04 01:38:17.800261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.800160 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" event={"ID":"ccbd0d38-f9ee-40d5-af7a-0688d085a961","Type":"ContainerStarted","Data":"87eaf130179eedfa0a71d8392b48e1bf46a1596255499b297f7adff97089bb9c"} Nov 04 01:38:17.818959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:17.818906 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" podStartSLOduration=1.81889062 podStartE2EDuration="1.81889062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:38:17.818439786 +0000 UTC m=+2129.841371796" watchObservedRunningTime="2026-11-04 01:38:17.81889062 +0000 UTC m=+2129.841822627" Nov 04 01:38:18.106990 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:18.106950 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3526d040-0df1-4795-8a36-6aea0141f3ef" path="/var/lib/kubelet/pods/3526d040-0df1-4795-8a36-6aea0141f3ef/volumes" Nov 04 01:38:18.806926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:18.806891 4312 generic.go:356] "Generic (PLEG): container finished" podID="ccbd0d38-f9ee-40d5-af7a-0688d085a961" containerID="d2a7d98f458652c248e5f9b1dc8cf57afa5a900bee77d6ad9dca2d5afd2f54f0" exitCode=0 Nov 04 01:38:18.807134 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:18.806984 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" event={"ID":"ccbd0d38-f9ee-40d5-af7a-0688d085a961","Type":"ContainerDied","Data":"d2a7d98f458652c248e5f9b1dc8cf57afa5a900bee77d6ad9dca2d5afd2f54f0"} Nov 04 01:38:19.940434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:19.940412 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:20.065891 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.065807 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbd0d38-f9ee-40d5-af7a-0688d085a961-kube-api-access-kcnbm\" (UniqueName: \"kubernetes.io/projected/ccbd0d38-f9ee-40d5-af7a-0688d085a961-kube-api-access-kcnbm\") pod \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " Nov 04 01:38:20.066042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.065907 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-openshift-ca\") pod \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " Nov 04 01:38:20.066042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.065980 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-xks-ca\") pod \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\" (UID: \"ccbd0d38-f9ee-40d5-af7a-0688d085a961\") " Nov 04 01:38:20.066329 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.066300 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-openshift-ca" pod "ccbd0d38-f9ee-40d5-af7a-0688d085a961" (UID: "ccbd0d38-f9ee-40d5-af7a-0688d085a961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:20.067987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.067958 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-xks-ca" pod "ccbd0d38-f9ee-40d5-af7a-0688d085a961" (UID: "ccbd0d38-f9ee-40d5-af7a-0688d085a961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:20.068071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.067985 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbd0d38-f9ee-40d5-af7a-0688d085a961-kube-api-access-kcnbm" pod "ccbd0d38-f9ee-40d5-af7a-0688d085a961" (UID: "ccbd0d38-f9ee-40d5-af7a-0688d085a961"). InnerVolumeSpecName "kube-api-access-kcnbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:20.167266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.167232 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:20.167266 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.167261 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnbm\" (UniqueName: \"kubernetes.io/projected/ccbd0d38-f9ee-40d5-af7a-0688d085a961-kube-api-access-kcnbm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:20.167495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.167276 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbd0d38-f9ee-40d5-af7a-0688d085a961-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:20.818737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.818709 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" Nov 04 01:38:20.818915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.818707 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz" event={"ID":"ccbd0d38-f9ee-40d5-af7a-0688d085a961","Type":"ContainerDied","Data":"87eaf130179eedfa0a71d8392b48e1bf46a1596255499b297f7adff97089bb9c"} Nov 04 01:38:20.818915 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:20.818833 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87eaf130179eedfa0a71d8392b48e1bf46a1596255499b297f7adff97089bb9c" Nov 04 01:38:29.457157 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:38:29.457120 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:38:39.631873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.631838 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5"] Nov 04 01:38:39.632290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.632275 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ccbd0d38-f9ee-40d5-af7a-0688d085a961" containerName="revoke-keys" Nov 04 01:38:39.632352 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.632291 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbd0d38-f9ee-40d5-af7a-0688d085a961" containerName="revoke-keys" Nov 04 01:38:39.632431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.632419 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbd0d38-f9ee-40d5-af7a-0688d085a961" containerName="revoke-keys" Nov 04 01:38:39.635948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.635930 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.641721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.641683 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:39.644502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.644454 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:38:39.644687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.644518 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:38:39.644687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.644546 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:38:39.646167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.646146 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5"] Nov 04 01:38:39.710197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.710170 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.710392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.710206 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.710392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.710250 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7w8w\" (UniqueName: \"kubernetes.io/projected/42fadec4-9d24-457c-950f-df2c156cb7bc-kube-api-access-h7w8w\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.811399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.811371 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.811558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.811407 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.811558 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.811449 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w8w\" (UniqueName: \"kubernetes.io/projected/42fadec4-9d24-457c-950f-df2c156cb7bc-kube-api-access-h7w8w\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.812069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.812046 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.813676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.813657 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.825870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.825838 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7w8w\" (UniqueName: \"kubernetes.io/projected/42fadec4-9d24-457c-950f-df2c156cb7bc-kube-api-access-h7w8w\") pod \"maas-api-revoke-sub-e2fc9c5b-p62f5\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:39.951724 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:39.951655 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:40.228098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:40.228058 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5"] Nov 04 01:38:40.229280 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:38:40.229259 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fadec4_9d24_457c_950f_df2c156cb7bc.slice/crio-527921ffd6c602b6abf30eb5f41ad0343334f9305c03c098fe1624d0f7da29f3 WatchSource:0}: Error finding container 527921ffd6c602b6abf30eb5f41ad0343334f9305c03c098fe1624d0f7da29f3: Status 404 returned error can't find the container with id 527921ffd6c602b6abf30eb5f41ad0343334f9305c03c098fe1624d0f7da29f3 Nov 04 01:38:40.948366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:40.948252 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" event={"ID":"42fadec4-9d24-457c-950f-df2c156cb7bc","Type":"ContainerStarted","Data":"73d1425776334914642ce15cb46699be4d2d5307e06fbb273442d737dd606a16"} Nov 04 01:38:40.948366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:40.948321 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" event={"ID":"42fadec4-9d24-457c-950f-df2c156cb7bc","Type":"ContainerStarted","Data":"527921ffd6c602b6abf30eb5f41ad0343334f9305c03c098fe1624d0f7da29f3"} Nov 04 01:38:40.968040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:40.967989 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" podStartSLOduration=1.9679754109999998 podStartE2EDuration="1.967975411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:38:40.966478593 +0000 UTC m=+2152.989410618" watchObservedRunningTime="2026-11-04 01:38:40.967975411 +0000 UTC m=+2152.990907418" Nov 04 01:38:41.101472 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:38:41.101432 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fadec4_9d24_457c_950f_df2c156cb7bc.slice/crio-73d1425776334914642ce15cb46699be4d2d5307e06fbb273442d737dd606a16.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fadec4_9d24_457c_950f_df2c156cb7bc.slice/crio-conmon-73d1425776334914642ce15cb46699be4d2d5307e06fbb273442d737dd606a16.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:38:41.962440 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:41.962399 4312 generic.go:356] "Generic (PLEG): container finished" podID="42fadec4-9d24-457c-950f-df2c156cb7bc" containerID="73d1425776334914642ce15cb46699be4d2d5307e06fbb273442d737dd606a16" exitCode=0 Nov 04 01:38:41.962797 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:41.962473 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" event={"ID":"42fadec4-9d24-457c-950f-df2c156cb7bc","Type":"ContainerDied","Data":"73d1425776334914642ce15cb46699be4d2d5307e06fbb273442d737dd606a16"} Nov 04 01:38:43.026517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.026482 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z"] Nov 04 01:38:43.030182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.030156 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11809375-c9p4z"] Nov 04 01:38:43.108235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.108211 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:43.144957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.144934 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fadec4-9d24-457c-950f-df2c156cb7bc-kube-api-access-h7w8w\" (UniqueName: \"kubernetes.io/projected/42fadec4-9d24-457c-950f-df2c156cb7bc-kube-api-access-h7w8w\") pod \"42fadec4-9d24-457c-950f-df2c156cb7bc\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " Nov 04 01:38:43.145102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.144991 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-xks-ca\") pod \"42fadec4-9d24-457c-950f-df2c156cb7bc\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " Nov 04 01:38:43.145102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.145041 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-openshift-ca\") pod \"42fadec4-9d24-457c-950f-df2c156cb7bc\" (UID: \"42fadec4-9d24-457c-950f-df2c156cb7bc\") " Nov 04 01:38:43.145601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.145496 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-openshift-ca" pod "42fadec4-9d24-457c-950f-df2c156cb7bc" (UID: "42fadec4-9d24-457c-950f-df2c156cb7bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:43.147147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.147127 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-xks-ca" pod "42fadec4-9d24-457c-950f-df2c156cb7bc" (UID: "42fadec4-9d24-457c-950f-df2c156cb7bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:43.147568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.147552 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fadec4-9d24-457c-950f-df2c156cb7bc-kube-api-access-h7w8w" pod "42fadec4-9d24-457c-950f-df2c156cb7bc" (UID: "42fadec4-9d24-457c-950f-df2c156cb7bc"). InnerVolumeSpecName "kube-api-access-h7w8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:43.246418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.246361 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7w8w\" (UniqueName: \"kubernetes.io/projected/42fadec4-9d24-457c-950f-df2c156cb7bc-kube-api-access-h7w8w\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:43.246418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.246383 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:43.246418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.246393 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fadec4-9d24-457c-950f-df2c156cb7bc-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:43.975174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.975140 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" Nov 04 01:38:43.975174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.975163 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5" event={"ID":"42fadec4-9d24-457c-950f-df2c156cb7bc","Type":"ContainerDied","Data":"527921ffd6c602b6abf30eb5f41ad0343334f9305c03c098fe1624d0f7da29f3"} Nov 04 01:38:43.975450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:43.975212 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="527921ffd6c602b6abf30eb5f41ad0343334f9305c03c098fe1624d0f7da29f3" Nov 04 01:38:44.107475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:44.107439 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03b246d-c8b8-458f-9fd6-c94f874bae32" path="/var/lib/kubelet/pods/a03b246d-c8b8-458f-9fd6-c94f874bae32/volumes" Nov 04 01:38:49.863054 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.863014 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp"] Nov 04 01:38:49.863914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.863506 4312 cpu_manager.go:414] "containerMap: removing container" podUID="42fadec4-9d24-457c-950f-df2c156cb7bc" containerName="revoke-keys" Nov 04 01:38:49.863914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.863521 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fadec4-9d24-457c-950f-df2c156cb7bc" containerName="revoke-keys" Nov 04 01:38:49.863914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.863597 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fadec4-9d24-457c-950f-df2c156cb7bc" containerName="revoke-keys" Nov 04 01:38:49.866814 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.866788 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:49.871068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.871042 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:38:49.872312 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.872289 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:49.873030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.872313 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:38:49.873178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.872382 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:38:49.876369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.876346 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp"] Nov 04 01:38:49.912072 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.912019 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw94k\" (UniqueName: \"kubernetes.io/projected/2562462f-1916-4aa4-960b-51c804005ee7-kube-api-access-jw94k\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:49.912281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.912106 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:49.912281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:49.912175 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.013848 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.013812 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.014074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.013908 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw94k\" (UniqueName: \"kubernetes.io/projected/2562462f-1916-4aa4-960b-51c804005ee7-kube-api-access-jw94k\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.014074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.013941 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.014669 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.014643 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.016345 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.016316 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.026269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.026241 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw94k\" (UniqueName: \"kubernetes.io/projected/2562462f-1916-4aa4-960b-51c804005ee7-kube-api-access-jw94k\") pod \"maas-api-revoke-sub-5a29efb3-k2jdp\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.183421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.183292 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:50.459560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:50.459530 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp"] Nov 04 01:38:50.460831 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:38:50.460800 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2562462f_1916_4aa4_960b_51c804005ee7.slice/crio-2acc29ee68dba2d62e3cb9d4c67607464fc9e711932f68f8b10bc764c80c5544 WatchSource:0}: Error finding container 2acc29ee68dba2d62e3cb9d4c67607464fc9e711932f68f8b10bc764c80c5544: Status 404 returned error can't find the container with id 2acc29ee68dba2d62e3cb9d4c67607464fc9e711932f68f8b10bc764c80c5544 Nov 04 01:38:51.015717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:51.015599 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" event={"ID":"2562462f-1916-4aa4-960b-51c804005ee7","Type":"ContainerStarted","Data":"1e3e9a7d5774f0bf77125e335c0cba8309d7f0d762644c650f1267079944ae95"} Nov 04 01:38:51.015717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:51.015681 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" event={"ID":"2562462f-1916-4aa4-960b-51c804005ee7","Type":"ContainerStarted","Data":"2acc29ee68dba2d62e3cb9d4c67607464fc9e711932f68f8b10bc764c80c5544"} Nov 04 01:38:51.035362 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:51.035282 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" podStartSLOduration=2.035264486 podStartE2EDuration="2.035264486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:38:51.034640501 +0000 UTC m=+2163.057572513" watchObservedRunningTime="2026-11-04 01:38:51.035264486 +0000 UTC m=+2163.058196494" Nov 04 01:38:52.040214 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:52.040167 4312 generic.go:356] "Generic (PLEG): container finished" podID="2562462f-1916-4aa4-960b-51c804005ee7" containerID="1e3e9a7d5774f0bf77125e335c0cba8309d7f0d762644c650f1267079944ae95" exitCode=0 Nov 04 01:38:52.041407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:52.041375 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" event={"ID":"2562462f-1916-4aa4-960b-51c804005ee7","Type":"ContainerDied","Data":"1e3e9a7d5774f0bf77125e335c0cba8309d7f0d762644c650f1267079944ae95"} Nov 04 01:38:53.757644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.757610 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:53.872371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.870796 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2562462f-1916-4aa4-960b-51c804005ee7-kube-api-access-jw94k\" (UniqueName: \"kubernetes.io/projected/2562462f-1916-4aa4-960b-51c804005ee7-kube-api-access-jw94k\") pod \"2562462f-1916-4aa4-960b-51c804005ee7\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " Nov 04 01:38:53.872371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.870887 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-xks-ca\") pod \"2562462f-1916-4aa4-960b-51c804005ee7\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " Nov 04 01:38:53.872371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.870985 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-openshift-ca\") pod \"2562462f-1916-4aa4-960b-51c804005ee7\" (UID: \"2562462f-1916-4aa4-960b-51c804005ee7\") " Nov 04 01:38:53.881992 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.881948 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-openshift-ca" pod "2562462f-1916-4aa4-960b-51c804005ee7" (UID: "2562462f-1916-4aa4-960b-51c804005ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:53.886470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.886419 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-xks-ca" pod "2562462f-1916-4aa4-960b-51c804005ee7" (UID: "2562462f-1916-4aa4-960b-51c804005ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:53.927993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.927923 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2562462f-1916-4aa4-960b-51c804005ee7-kube-api-access-jw94k" pod "2562462f-1916-4aa4-960b-51c804005ee7" (UID: "2562462f-1916-4aa4-960b-51c804005ee7"). InnerVolumeSpecName "kube-api-access-jw94k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:53.972207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.972103 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:53.972207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.972151 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2562462f-1916-4aa4-960b-51c804005ee7-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:53.972207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:53.972171 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw94k\" (UniqueName: \"kubernetes.io/projected/2562462f-1916-4aa4-960b-51c804005ee7-kube-api-access-jw94k\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:54.068195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:54.068159 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" Nov 04 01:38:54.068195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:54.068177 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp" event={"ID":"2562462f-1916-4aa4-960b-51c804005ee7","Type":"ContainerDied","Data":"2acc29ee68dba2d62e3cb9d4c67607464fc9e711932f68f8b10bc764c80c5544"} Nov 04 01:38:54.068434 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:54.068227 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2acc29ee68dba2d62e3cb9d4c67607464fc9e711932f68f8b10bc764c80c5544" Nov 04 01:38:55.024471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:55.024430 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr"] Nov 04 01:38:55.027702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:55.027676 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191d967d-7vbrr"] Nov 04 01:38:56.106471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:38:56.106438 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1008bfc8-a10c-4ce0-9fed-2d5f1d5da880" path="/var/lib/kubelet/pods/1008bfc8-a10c-4ce0-9fed-2d5f1d5da880/volumes" Nov 04 01:38:59.456613 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:38:59.456576 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:39:05.362361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.362251 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz"] Nov 04 01:39:05.365548 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.365522 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hl8vz"] Nov 04 01:39:05.384425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.384399 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj"] Nov 04 01:39:05.384874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.384856 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2562462f-1916-4aa4-960b-51c804005ee7" containerName="revoke-keys" Nov 04 01:39:05.384986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.384877 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2562462f-1916-4aa4-960b-51c804005ee7" containerName="revoke-keys" Nov 04 01:39:05.384986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.384977 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2562462f-1916-4aa4-960b-51c804005ee7" containerName="revoke-keys" Nov 04 01:39:05.388513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.388494 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.392276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.392250 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:39:05.392476 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.392457 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:39:05.392613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.392513 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:05.392613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.392573 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:39:05.399435 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.399413 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj"] Nov 04 01:39:05.488276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.488241 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.488452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.488299 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5ww\" (UniqueName: \"kubernetes.io/projected/ab27f145-2165-428f-a3b3-512c70ff2320-kube-api-access-ts5ww\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.488452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.488421 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.589116 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.589084 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.589300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.589135 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5ww\" (UniqueName: \"kubernetes.io/projected/ab27f145-2165-428f-a3b3-512c70ff2320-kube-api-access-ts5ww\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.589300 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.589157 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.589838 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.589815 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.591478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.591449 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.601650 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.601624 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5ww\" (UniqueName: \"kubernetes.io/projected/ab27f145-2165-428f-a3b3-512c70ff2320-kube-api-access-ts5ww\") pod \"maas-api-revoke-sub-3d4156f1-4s4cj\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.706615 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.706539 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:05.981828 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:05.981792 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj"] Nov 04 01:39:05.983262 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:39:05.983234 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab27f145_2165_428f_a3b3_512c70ff2320.slice/crio-a929abc85b42eb62d2ffb76ff17588c1531ccd4f3b3c6f022da8395f6a6e76da WatchSource:0}: Error finding container a929abc85b42eb62d2ffb76ff17588c1531ccd4f3b3c6f022da8395f6a6e76da: Status 404 returned error can't find the container with id a929abc85b42eb62d2ffb76ff17588c1531ccd4f3b3c6f022da8395f6a6e76da Nov 04 01:39:06.108257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:06.108231 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbd0d38-f9ee-40d5-af7a-0688d085a961" path="/var/lib/kubelet/pods/ccbd0d38-f9ee-40d5-af7a-0688d085a961/volumes" Nov 04 01:39:06.134313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:06.134288 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" event={"ID":"ab27f145-2165-428f-a3b3-512c70ff2320","Type":"ContainerStarted","Data":"a929abc85b42eb62d2ffb76ff17588c1531ccd4f3b3c6f022da8395f6a6e76da"} Nov 04 01:39:07.141381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:07.141329 4312 generic.go:356] "Generic (PLEG): container finished" podID="ab27f145-2165-428f-a3b3-512c70ff2320" containerID="c09a6457b63eaa2b12f4df1cc60962c90e4a130821eb64294066995e154b1c39" exitCode=0 Nov 04 01:39:07.141799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:07.141433 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" event={"ID":"ab27f145-2165-428f-a3b3-512c70ff2320","Type":"ContainerDied","Data":"c09a6457b63eaa2b12f4df1cc60962c90e4a130821eb64294066995e154b1c39"} Nov 04 01:39:08.277563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.277541 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:08.415733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.415652 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab27f145-2165-428f-a3b3-512c70ff2320-kube-api-access-ts5ww\" (UniqueName: \"kubernetes.io/projected/ab27f145-2165-428f-a3b3-512c70ff2320-kube-api-access-ts5ww\") pod \"ab27f145-2165-428f-a3b3-512c70ff2320\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " Nov 04 01:39:08.415890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.415745 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-xks-ca\") pod \"ab27f145-2165-428f-a3b3-512c70ff2320\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " Nov 04 01:39:08.415976 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.415948 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-openshift-ca\") pod \"ab27f145-2165-428f-a3b3-512c70ff2320\" (UID: \"ab27f145-2165-428f-a3b3-512c70ff2320\") " Nov 04 01:39:08.416277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.416251 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-openshift-ca" pod "ab27f145-2165-428f-a3b3-512c70ff2320" (UID: "ab27f145-2165-428f-a3b3-512c70ff2320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:08.416452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.416430 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:08.417818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.417794 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab27f145-2165-428f-a3b3-512c70ff2320-kube-api-access-ts5ww" pod "ab27f145-2165-428f-a3b3-512c70ff2320" (UID: "ab27f145-2165-428f-a3b3-512c70ff2320"). InnerVolumeSpecName "kube-api-access-ts5ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:08.417818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.417796 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-xks-ca" pod "ab27f145-2165-428f-a3b3-512c70ff2320" (UID: "ab27f145-2165-428f-a3b3-512c70ff2320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:08.517208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.517179 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5ww\" (UniqueName: \"kubernetes.io/projected/ab27f145-2165-428f-a3b3-512c70ff2320-kube-api-access-ts5ww\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:08.517208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:08.517205 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab27f145-2165-428f-a3b3-512c70ff2320-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:09.154924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:09.154896 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" Nov 04 01:39:09.155104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:09.154897 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj" event={"ID":"ab27f145-2165-428f-a3b3-512c70ff2320","Type":"ContainerDied","Data":"a929abc85b42eb62d2ffb76ff17588c1531ccd4f3b3c6f022da8395f6a6e76da"} Nov 04 01:39:09.155104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:09.155019 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a929abc85b42eb62d2ffb76ff17588c1531ccd4f3b3c6f022da8395f6a6e76da" Nov 04 01:39:20.859282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.859233 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c"] Nov 04 01:39:20.860033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.859794 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ab27f145-2165-428f-a3b3-512c70ff2320" containerName="revoke-keys" Nov 04 01:39:20.860033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.859814 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab27f145-2165-428f-a3b3-512c70ff2320" containerName="revoke-keys" Nov 04 01:39:20.860033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.859928 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab27f145-2165-428f-a3b3-512c70ff2320" containerName="revoke-keys" Nov 04 01:39:20.862925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.862907 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.869536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.869517 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-042372-openshift-default-dockercfg-prwmj\"" Nov 04 01:39:20.878524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.878498 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c"] Nov 04 01:39:20.920834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.920799 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-istio-token\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.920834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.920839 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-envoy\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.921040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.920856 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61130f1c-058d-459a-a016-08a54774ca00-istiod-ca-cert\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.921040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.920989 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61130f1c-058d-459a-a016-08a54774ca00-istio-podinfo\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.921040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.921006 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-credential-socket\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.921147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.921085 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-socket\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.921147 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.921123 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-data\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.921216 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.921144 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-certs\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:20.921255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:20.921228 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfg9\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-kube-api-access-zkfg9\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022437 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-data\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022474 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-certs\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022500 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfg9\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-kube-api-access-zkfg9\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022548 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-istio-token\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022571 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-envoy\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022589 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61130f1c-058d-459a-a016-08a54774ca00-istiod-ca-cert\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022656 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61130f1c-058d-459a-a016-08a54774ca00-istio-podinfo\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022685 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-credential-socket\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.022946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.022757 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-socket\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.023947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.023002 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-certs\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.023947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.023271 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-socket\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.023947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.023483 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-data\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.023947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.023532 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61130f1c-058d-459a-a016-08a54774ca00-istiod-ca-cert\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.023947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.023896 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-credential-socket\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.025489 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.025444 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-envoy\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.026593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.026500 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61130f1c-058d-459a-a016-08a54774ca00-istio-podinfo\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.035739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.035689 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfg9\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-kube-api-access-zkfg9\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.036899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.036872 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-istio-token\") pod \"e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.177369 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.177268 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:21.465716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.465689 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c"] Nov 04 01:39:21.466634 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:39:21.466608 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61130f1c_058d_459a_a016_08a54774ca00.slice/crio-edd0dfe084343a1c06e49c53e66bd38cc7626661d7f88c309d283e6e45698ec8 WatchSource:0}: Error finding container edd0dfe084343a1c06e49c53e66bd38cc7626661d7f88c309d283e6e45698ec8: Status 404 returned error can't find the container with id edd0dfe084343a1c06e49c53e66bd38cc7626661d7f88c309d283e6e45698ec8 Nov 04 01:39:21.468899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.468859 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:39:21.468988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.468957 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:39:21.469038 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:21.469006 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:39:22.259648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:22.259606 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" event={"ID":"61130f1c-058d-459a-a016-08a54774ca00","Type":"ContainerStarted","Data":"3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b"} Nov 04 01:39:22.260059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:22.259665 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" event={"ID":"61130f1c-058d-459a-a016-08a54774ca00","Type":"ContainerStarted","Data":"edd0dfe084343a1c06e49c53e66bd38cc7626661d7f88c309d283e6e45698ec8"} Nov 04 01:39:22.287175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:22.287122 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" podStartSLOduration=2.287108098 podStartE2EDuration="2.287108098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:22.283677206 +0000 UTC m=+2194.306609213" watchObservedRunningTime="2026-11-04 01:39:22.287108098 +0000 UTC m=+2194.310040104" Nov 04 01:39:23.177689 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:23.177659 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:23.182759 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:23.182731 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:23.265518 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:23.265486 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:23.266251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:23.266232 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:25.763070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.763033 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj"] Nov 04 01:39:25.767126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.767092 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.773833 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.773800 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:39:25.773981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.773963 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:39:25.786536 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.786508 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj"] Nov 04 01:39:25.788343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.788293 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-042372-serving-cert\"" Nov 04 01:39:25.788343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.788316 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rbkg2\"" Nov 04 01:39:25.788538 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.788442 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:39:25.850709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.850677 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6"] Nov 04 01:39:25.854990 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.854970 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:25.868808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.868783 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzxh\" (UniqueName: \"kubernetes.io/projected/901fe8b0-4ef4-4d90-898f-0834ee9b022a-kube-api-access-gtzxh\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.868968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.868902 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901fe8b0-4ef4-4d90-898f-0834ee9b022a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.868968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.868959 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.869093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.869021 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.872412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.872388 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6"] Nov 04 01:39:25.872551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.872452 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-042372\"" Nov 04 01:39:25.886494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.886472 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd"] Nov 04 01:39:25.890152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.890134 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:25.890637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.890619 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8d98d1ab-dockercfg-wcwm5\"" Nov 04 01:39:25.906510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.906488 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd"] Nov 04 01:39:25.970325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970284 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dec1e7a-65d8-4940-94ac-d54178abc8b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:25.970554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970402 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzxh\" (UniqueName: \"kubernetes.io/projected/901fe8b0-4ef4-4d90-898f-0834ee9b022a-kube-api-access-gtzxh\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.970554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970450 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901fe8b0-4ef4-4d90-898f-0834ee9b022a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.970554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970479 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.970554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970503 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17194cf9-23f2-4233-a166-2b7a73a00825-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042372-745659844tvhpd\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:25.970554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970541 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhm9s\" (UniqueName: \"kubernetes.io/projected/17194cf9-23f2-4233-a166-2b7a73a00825-kube-api-access-xhm9s\") pod \"payload-processing-e2e-worker-single-main-042372-745659844tvhpd\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:25.970864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970570 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.970864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.970595 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gh4c\" (UniqueName: \"kubernetes.io/projected/9dec1e7a-65d8-4940-94ac-d54178abc8b9-kube-api-access-8gh4c\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:25.971641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.971606 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901fe8b0-4ef4-4d90-898f-0834ee9b022a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.976970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.976941 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:25.977468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.977420 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:26.003379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:25.996741 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzxh\" (UniqueName: \"kubernetes.io/projected/901fe8b0-4ef4-4d90-898f-0834ee9b022a-kube-api-access-gtzxh\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:26.071730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.071696 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17194cf9-23f2-4233-a166-2b7a73a00825-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042372-745659844tvhpd\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:26.071936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.071743 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm9s\" (UniqueName: \"kubernetes.io/projected/17194cf9-23f2-4233-a166-2b7a73a00825-kube-api-access-xhm9s\") pod \"payload-processing-e2e-worker-single-main-042372-745659844tvhpd\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:26.071936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.071785 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gh4c\" (UniqueName: \"kubernetes.io/projected/9dec1e7a-65d8-4940-94ac-d54178abc8b9-kube-api-access-8gh4c\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:26.071936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.071819 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dec1e7a-65d8-4940-94ac-d54178abc8b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:26.072655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.072626 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17194cf9-23f2-4233-a166-2b7a73a00825-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042372-745659844tvhpd\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:26.072712 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.072652 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dec1e7a-65d8-4940-94ac-d54178abc8b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:26.083711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.083688 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:26.086495 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.086476 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhm9s\" (UniqueName: \"kubernetes.io/projected/17194cf9-23f2-4233-a166-2b7a73a00825-kube-api-access-xhm9s\") pod \"payload-processing-e2e-worker-single-main-042372-745659844tvhpd\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:26.088473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.088439 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gh4c\" (UniqueName: \"kubernetes.io/projected/9dec1e7a-65d8-4940-94ac-d54178abc8b9-kube-api-access-8gh4c\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:26.169060 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.168776 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:26.205441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.205416 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:26.369080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.369054 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj"] Nov 04 01:39:26.370437 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:39:26.370410 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901fe8b0_4ef4_4d90_898f_0834ee9b022a.slice/crio-920e4ea9f56f3b28184d6ab6ee32aba3ad50cd084c42689bbff4d6bb653e1f60 WatchSource:0}: Error finding container 920e4ea9f56f3b28184d6ab6ee32aba3ad50cd084c42689bbff4d6bb653e1f60: Status 404 returned error can't find the container with id 920e4ea9f56f3b28184d6ab6ee32aba3ad50cd084c42689bbff4d6bb653e1f60 Nov 04 01:39:26.449863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.449837 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6"] Nov 04 01:39:26.450639 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:39:26.450606 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dec1e7a_65d8_4940_94ac_d54178abc8b9.slice/crio-12cc7c92d7c45ded3e536ceacf1d7766ccd5fcd4998eb0f26d1333e2bd41f2ae WatchSource:0}: Error finding container 12cc7c92d7c45ded3e536ceacf1d7766ccd5fcd4998eb0f26d1333e2bd41f2ae: Status 404 returned error can't find the container with id 12cc7c92d7c45ded3e536ceacf1d7766ccd5fcd4998eb0f26d1333e2bd41f2ae Nov 04 01:39:26.499865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:26.499844 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd"] Nov 04 01:39:26.501807 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:39:26.501776 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17194cf9_23f2_4233_a166_2b7a73a00825.slice/crio-470a0c7f20280d48f56d2609550c77e2daf5fce1903dd83e07c1d787f46f3b21 WatchSource:0}: Error finding container 470a0c7f20280d48f56d2609550c77e2daf5fce1903dd83e07c1d787f46f3b21: Status 404 returned error can't find the container with id 470a0c7f20280d48f56d2609550c77e2daf5fce1903dd83e07c1d787f46f3b21 Nov 04 01:39:27.292239 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.292202 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" event={"ID":"17194cf9-23f2-4233-a166-2b7a73a00825","Type":"ContainerStarted","Data":"2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe"} Nov 04 01:39:27.292738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.292264 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" event={"ID":"17194cf9-23f2-4233-a166-2b7a73a00825","Type":"ContainerStarted","Data":"470a0c7f20280d48f56d2609550c77e2daf5fce1903dd83e07c1d787f46f3b21"} Nov 04 01:39:27.292738 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.292354 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:27.293810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.293785 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" event={"ID":"901fe8b0-4ef4-4d90-898f-0834ee9b022a","Type":"ContainerStarted","Data":"2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b"} Nov 04 01:39:27.293948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.293830 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" event={"ID":"901fe8b0-4ef4-4d90-898f-0834ee9b022a","Type":"ContainerStarted","Data":"920e4ea9f56f3b28184d6ab6ee32aba3ad50cd084c42689bbff4d6bb653e1f60"} Nov 04 01:39:27.293948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.293858 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:27.295445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.295421 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" event={"ID":"9dec1e7a-65d8-4940-94ac-d54178abc8b9","Type":"ContainerStarted","Data":"8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c"} Nov 04 01:39:27.295578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.295462 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" event={"ID":"9dec1e7a-65d8-4940-94ac-d54178abc8b9","Type":"ContainerStarted","Data":"12cc7c92d7c45ded3e536ceacf1d7766ccd5fcd4998eb0f26d1333e2bd41f2ae"} Nov 04 01:39:27.295578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.295555 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:27.315359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.315303 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" podStartSLOduration=2.315290178 podStartE2EDuration="2.315290178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:27.313095 +0000 UTC m=+2199.336027010" watchObservedRunningTime="2026-11-04 01:39:27.315290178 +0000 UTC m=+2199.338222204" Nov 04 01:39:27.338790 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.338749 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" podStartSLOduration=2.338736873 podStartE2EDuration="2.338736873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:27.336219275 +0000 UTC m=+2199.359151282" watchObservedRunningTime="2026-11-04 01:39:27.338736873 +0000 UTC m=+2199.361668877" Nov 04 01:39:27.370467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:27.370426 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" podStartSLOduration=1.836719907 podStartE2EDuration="2.370413259s" totalImagesPullingTime="533.693352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:27.369349375 +0000 UTC m=+2199.392281404" watchObservedRunningTime="2026-11-04 01:39:27.370413259 +0000 UTC m=+2199.393345265" Nov 04 01:39:29.481458 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:39:29.481372 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:39:33.307627 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:33.307591 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:39:37.355120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:37.355087 4312 generic.go:356] "Generic (PLEG): container finished" podID="61130f1c-058d-459a-a016-08a54774ca00" containerID="3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b" exitCode=0 Nov 04 01:39:37.355661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:37.355163 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" event={"ID":"61130f1c-058d-459a-a016-08a54774ca00","Type":"ContainerDied","Data":"3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b"} Nov 04 01:39:37.355661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:37.355571 4312 scope.go:122] "RemoveContainer" containerID="3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b" Nov 04 01:39:37.356818 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:37.356791 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:39:37.356882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:37.356868 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:39:37.356920 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:37.356909 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:39:38.269355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:38.267452 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:38.303911 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:38.303882 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:39:38.304119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:38.303955 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:39:38.385910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:38.385863 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" event={"ID":"61130f1c-058d-459a-a016-08a54774ca00","Type":"ContainerStarted","Data":"cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb"} Nov 04 01:39:39.177876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:39.177843 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:39.179703 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:39.179669 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 01:39:39.179880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:39.179729 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 01:39:40.178564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:40.178528 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 01:39:40.179044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:40.178591 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 01:39:41.177894 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.177852 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 01:39:41.178108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.177915 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 01:39:41.890618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.890587 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9"] Nov 04 01:39:41.894676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.894654 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:41.903944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.903826 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-042372\"/\"openshift-service-ca.crt\"" Nov 04 01:39:41.903944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.903918 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-042372\"/\"default-dockercfg-tsq56\"" Nov 04 01:39:41.905099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.905069 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9"] Nov 04 01:39:41.943576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.943551 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-042372\"/\"simulator-single-main-042372-kserve-self-signed-certs\"" Nov 04 01:39:41.975880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.975848 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tmp-dir\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:41.976053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.975959 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fv4\" (UniqueName: \"kubernetes.io/projected/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-kube-api-access-m6fv4\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:41.976053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.976034 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-dshm\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:41.976197 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.976161 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-model-cache\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:41.976255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.976203 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-home\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:41.976255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:41.976229 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tls-certs\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.004217 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.004193 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-042372\"/\"kube-root-ca.crt\"" Nov 04 01:39:42.077883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.077850 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-home\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.078582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.078552 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tls-certs\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.078735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.078614 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tmp-dir\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.078735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.078659 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fv4\" (UniqueName: \"kubernetes.io/projected/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-kube-api-access-m6fv4\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.078735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.078686 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-dshm\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.078937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.078800 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-model-cache\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.079088 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.079068 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-model-cache\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.079240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.078481 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-home\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.079240 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.079179 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tmp-dir\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.081280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.081249 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-dshm\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.081549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.081529 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tls-certs\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.091800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.091773 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fv4\" (UniqueName: \"kubernetes.io/projected/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-kube-api-access-m6fv4\") pod \"simulator-single-main-042372-kserve-7fc56f5db8-s68d9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.177923 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.177831 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 01:39:42.178291 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.178259 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 01:39:42.214626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.214595 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:42.514736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:42.514709 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9"] Nov 04 01:39:42.515762 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:39:42.515731 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6aee319_7f4d_4d2c_b60c_a4240bb183e9.slice/crio-fa093b1df8d5010e3df7462c9c2ae4483ea96a7c8c79465a3e26459eb0e7344f WatchSource:0}: Error finding container fa093b1df8d5010e3df7462c9c2ae4483ea96a7c8c79465a3e26459eb0e7344f: Status 404 returned error can't find the container with id fa093b1df8d5010e3df7462c9c2ae4483ea96a7c8c79465a3e26459eb0e7344f Nov 04 01:39:43.178167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:43.178132 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 04 01:39:43.178601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:43.178189 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 04 01:39:43.420137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:43.420104 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" event={"ID":"f6aee319-7f4d-4d2c-b60c-a4240bb183e9","Type":"ContainerStarted","Data":"56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736"} Nov 04 01:39:43.420137 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:43.420147 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" event={"ID":"f6aee319-7f4d-4d2c-b60c-a4240bb183e9","Type":"ContainerStarted","Data":"fa093b1df8d5010e3df7462c9c2ae4483ea96a7c8c79465a3e26459eb0e7344f"} Nov 04 01:39:43.442245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:43.442159 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" podStartSLOduration=2.442145356 podStartE2EDuration="2.442145356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:43.439191876 +0000 UTC m=+2215.462123896" watchObservedRunningTime="2026-11-04 01:39:43.442145356 +0000 UTC m=+2215.465077363" Nov 04 01:39:44.215048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:44.215017 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:44.605425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:44.605392 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:44.605725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:44.605690 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:44.606477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:44.606454 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:39:50.583199 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:50.583167 4312 scope.go:122] "RemoveContainer" containerID="539db97a4031bf788ba54c5fe16a84d0c921e7a233443a9f030a7cee950c74e0" Nov 04 01:39:52.215467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:52.215433 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:52.227779 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:52.227753 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:52.504225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:52.504142 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:39:53.031020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:53.030930 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj"] Nov 04 01:39:53.034042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:53.034015 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2pzqj"] Nov 04 01:39:54.108951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:39:54.108916 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d40cf27-a4fb-4057-9b57-d829decae5f9" path="/var/lib/kubelet/pods/5d40cf27-a4fb-4057-9b57-d829decae5f9/volumes" Nov 04 01:39:59.458545 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:39:59.458505 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:40:27.652044 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:27.652008 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9"] Nov 04 01:40:29.458577 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:40:29.458541 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:40:29.498003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:29.497971 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd"] Nov 04 01:40:29.512007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:29.511976 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6"] Nov 04 01:40:30.604912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.604871 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj"] Nov 04 01:40:30.605454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.605185 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" podUID="901fe8b0-4ef4-4d90-898f-0834ee9b022a" containerName="maas-api" containerID="cri-o://2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b" gracePeriod=30 Nov 04 01:40:30.884100 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.884065 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:40:30.982794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.982753 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901fe8b0-4ef4-4d90-898f-0834ee9b022a-kube-api-access-gtzxh\" (UniqueName: \"kubernetes.io/projected/901fe8b0-4ef4-4d90-898f-0834ee9b022a-kube-api-access-gtzxh\") pod \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " Nov 04 01:40:30.983010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.982811 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-metrics-certs\") pod \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " Nov 04 01:40:30.983010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.982866 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901fe8b0-4ef4-4d90-898f-0834ee9b022a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901fe8b0-4ef4-4d90-898f-0834ee9b022a-openshift-service-ca\") pod \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " Nov 04 01:40:30.983010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.982910 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-maas-api-tls\") pod \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\" (UID: \"901fe8b0-4ef4-4d90-898f-0834ee9b022a\") " Nov 04 01:40:30.983314 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.983284 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901fe8b0-4ef4-4d90-898f-0834ee9b022a-openshift-service-ca" pod "901fe8b0-4ef4-4d90-898f-0834ee9b022a" (UID: "901fe8b0-4ef4-4d90-898f-0834ee9b022a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:30.985065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.985031 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-maas-api-tls" pod "901fe8b0-4ef4-4d90-898f-0834ee9b022a" (UID: "901fe8b0-4ef4-4d90-898f-0834ee9b022a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:30.985198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.985062 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-metrics-certs" pod "901fe8b0-4ef4-4d90-898f-0834ee9b022a" (UID: "901fe8b0-4ef4-4d90-898f-0834ee9b022a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:30.985198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:30.985120 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901fe8b0-4ef4-4d90-898f-0834ee9b022a-kube-api-access-gtzxh" pod "901fe8b0-4ef4-4d90-898f-0834ee9b022a" (UID: "901fe8b0-4ef4-4d90-898f-0834ee9b022a"). InnerVolumeSpecName "kube-api-access-gtzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:31.084224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.084192 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901fe8b0-4ef4-4d90-898f-0834ee9b022a-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:31.084224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.084221 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:31.084474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.084233 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzxh\" (UniqueName: \"kubernetes.io/projected/901fe8b0-4ef4-4d90-898f-0834ee9b022a-kube-api-access-gtzxh\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:31.084474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.084250 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/901fe8b0-4ef4-4d90-898f-0834ee9b022a-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:31.713420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.713384 4312 generic.go:356] "Generic (PLEG): container finished" podID="901fe8b0-4ef4-4d90-898f-0834ee9b022a" containerID="2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b" exitCode=0 Nov 04 01:40:31.713872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.713433 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" Nov 04 01:40:31.713872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.713470 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" event={"ID":"901fe8b0-4ef4-4d90-898f-0834ee9b022a","Type":"ContainerDied","Data":"2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b"} Nov 04 01:40:31.713872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.713523 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj" event={"ID":"901fe8b0-4ef4-4d90-898f-0834ee9b022a","Type":"ContainerDied","Data":"920e4ea9f56f3b28184d6ab6ee32aba3ad50cd084c42689bbff4d6bb653e1f60"} Nov 04 01:40:31.713872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.713545 4312 scope.go:122] "RemoveContainer" containerID="2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b" Nov 04 01:40:31.726481 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.726175 4312 scope.go:122] "RemoveContainer" containerID="2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b" Nov 04 01:40:31.726565 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:40:31.726535 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b\": container with ID starting with 2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b not found: ID does not exist" containerID="2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b" Nov 04 01:40:31.726629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.726560 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b"} err="failed to get container status \"2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b\": rpc error: code = NotFound desc = could not find container \"2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b\": container with ID starting with 2a8973a51be4984b7c58d22f28d79aab6e73b077decea79c9ba916cc338b9c9b not found: ID does not exist" Nov 04 01:40:31.737925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.737901 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj"] Nov 04 01:40:31.741771 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:31.741749 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-tk9cj"] Nov 04 01:40:32.107643 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:32.107604 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901fe8b0-4ef4-4d90-898f-0834ee9b022a" path="/var/lib/kubelet/pods/901fe8b0-4ef4-4d90-898f-0834ee9b022a/volumes" Nov 04 01:40:34.498982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:34.498936 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" podUID="17194cf9-23f2-4233-a166-2b7a73a00825" containerName="payload-processing" containerID="cri-o://2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe" gracePeriod=25 Nov 04 01:40:34.512822 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:34.512783 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" podUID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" containerName="payload-pre-processing" containerID="cri-o://8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c" gracePeriod=25 Nov 04 01:40:38.302439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:38.302396 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" podUID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:38.302909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:38.302395 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" podUID="17194cf9-23f2-4233-a166-2b7a73a00825" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:42.672589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:42.672538 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" podUID="f6aee319-7f4d-4d2c-b60c-a4240bb183e9" containerName="main" containerID="cri-o://56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736" gracePeriod=45 Nov 04 01:40:42.917474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:42.917447 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:40:43.007677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.007645 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-model-cache\") pod \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " Nov 04 01:40:43.007872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.007748 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-dshm\") pod \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " Nov 04 01:40:43.007872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.007826 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-home\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-home\") pod \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " Nov 04 01:40:43.007998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.007887 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-model-cache" pod "f6aee319-7f4d-4d2c-b60c-a4240bb183e9" (UID: "f6aee319-7f4d-4d2c-b60c-a4240bb183e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:43.007998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.007892 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-kube-api-access-m6fv4\" (UniqueName: \"kubernetes.io/projected/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-kube-api-access-m6fv4\") pod \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " Nov 04 01:40:43.007998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.007958 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tls-certs\") pod \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " Nov 04 01:40:43.008156 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.008097 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tmp-dir\") pod \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\" (UID: \"f6aee319-7f4d-4d2c-b60c-a4240bb183e9\") " Nov 04 01:40:43.008224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.008154 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-home" pod "f6aee319-7f4d-4d2c-b60c-a4240bb183e9" (UID: "f6aee319-7f4d-4d2c-b60c-a4240bb183e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:43.008393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.008366 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tmp-dir" pod "f6aee319-7f4d-4d2c-b60c-a4240bb183e9" (UID: "f6aee319-7f4d-4d2c-b60c-a4240bb183e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:43.008531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.008513 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:43.008603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.008538 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:43.008603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.008554 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:43.009951 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.009925 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-dshm" pod "f6aee319-7f4d-4d2c-b60c-a4240bb183e9" (UID: "f6aee319-7f4d-4d2c-b60c-a4240bb183e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:43.010185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.010161 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-kube-api-access-m6fv4" pod "f6aee319-7f4d-4d2c-b60c-a4240bb183e9" (UID: "f6aee319-7f4d-4d2c-b60c-a4240bb183e9"). InnerVolumeSpecName "kube-api-access-m6fv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:43.010265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.010246 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tls-certs" pod "f6aee319-7f4d-4d2c-b60c-a4240bb183e9" (UID: "f6aee319-7f4d-4d2c-b60c-a4240bb183e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:43.109563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.109535 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6fv4\" (UniqueName: \"kubernetes.io/projected/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-kube-api-access-m6fv4\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:43.109563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.109560 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:43.109563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.109569 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6aee319-7f4d-4d2c-b60c-a4240bb183e9-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:43.790002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.789966 4312 generic.go:356] "Generic (PLEG): container finished" podID="f6aee319-7f4d-4d2c-b60c-a4240bb183e9" containerID="56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736" exitCode=0 Nov 04 01:40:43.790466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.790009 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" Nov 04 01:40:43.790466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.790051 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" event={"ID":"f6aee319-7f4d-4d2c-b60c-a4240bb183e9","Type":"ContainerDied","Data":"56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736"} Nov 04 01:40:43.790466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.790097 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9" event={"ID":"f6aee319-7f4d-4d2c-b60c-a4240bb183e9","Type":"ContainerDied","Data":"fa093b1df8d5010e3df7462c9c2ae4483ea96a7c8c79465a3e26459eb0e7344f"} Nov 04 01:40:43.790466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.790117 4312 scope.go:122] "RemoveContainer" containerID="56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736" Nov 04 01:40:43.799676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.799651 4312 scope.go:122] "RemoveContainer" containerID="56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736" Nov 04 01:40:43.799933 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:40:43.799916 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736\": container with ID starting with 56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736 not found: ID does not exist" containerID="56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736" Nov 04 01:40:43.799994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.799940 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736"} err="failed to get container status \"56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736\": rpc error: code = NotFound desc = could not find container \"56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736\": container with ID starting with 56cd9c497c7ddef3c26ff0394535781e2857a7c23e24f5c4a5c0aed327919736 not found: ID does not exist" Nov 04 01:40:43.813125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.813099 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9"] Nov 04 01:40:43.815506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:43.815481 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-042372/simulator-single-main-042372-kserve-7fc56f5db8-s68d9"] Nov 04 01:40:44.107057 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:44.106966 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6aee319-7f4d-4d2c-b60c-a4240bb183e9" path="/var/lib/kubelet/pods/f6aee319-7f4d-4d2c-b60c-a4240bb183e9/volumes" Nov 04 01:40:46.025413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:46.025379 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw"] Nov 04 01:40:46.028810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:46.028786 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2vbhw"] Nov 04 01:40:46.107121 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:46.107095 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83540a58-81a7-41de-b896-cb38b0b0e3ce" path="/var/lib/kubelet/pods/83540a58-81a7-41de-b896-cb38b0b0e3ce/volumes" Nov 04 01:40:48.302583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:48.302546 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" podUID="17194cf9-23f2-4233-a166-2b7a73a00825" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:48.302963 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:48.302552 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" podUID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:50.607540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:50.607509 4312 scope.go:122] "RemoveContainer" containerID="733dc12458c0376dced644cc9cf2b7693582cbf25df6769491feb630aef81003" Nov 04 01:40:50.616596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:50.616575 4312 scope.go:122] "RemoveContainer" containerID="06846f3c142a0fc8147c0ac88612b7d4d513f7fe962d67007ffa5c3c9517513c" Nov 04 01:40:54.771786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.771761 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:40:54.795811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.795788 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:40:54.855397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.855365 4312 generic.go:356] "Generic (PLEG): container finished" podID="17194cf9-23f2-4233-a166-2b7a73a00825" containerID="2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe" exitCode=0 Nov 04 01:40:54.855591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.855406 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" Nov 04 01:40:54.855591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.855443 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" event={"ID":"17194cf9-23f2-4233-a166-2b7a73a00825","Type":"ContainerDied","Data":"2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe"} Nov 04 01:40:54.855591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.855496 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd" event={"ID":"17194cf9-23f2-4233-a166-2b7a73a00825","Type":"ContainerDied","Data":"470a0c7f20280d48f56d2609550c77e2daf5fce1903dd83e07c1d787f46f3b21"} Nov 04 01:40:54.855591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.855518 4312 scope.go:122] "RemoveContainer" containerID="2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe" Nov 04 01:40:54.856907 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.856887 4312 generic.go:356] "Generic (PLEG): container finished" podID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" containerID="8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c" exitCode=0 Nov 04 01:40:54.856996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.856934 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" Nov 04 01:40:54.856996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.856958 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" event={"ID":"9dec1e7a-65d8-4940-94ac-d54178abc8b9","Type":"ContainerDied","Data":"8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c"} Nov 04 01:40:54.857077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.856995 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6" event={"ID":"9dec1e7a-65d8-4940-94ac-d54178abc8b9","Type":"ContainerDied","Data":"12cc7c92d7c45ded3e536ceacf1d7766ccd5fcd4998eb0f26d1333e2bd41f2ae"} Nov 04 01:40:54.864702 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.864684 4312 scope.go:122] "RemoveContainer" containerID="2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe" Nov 04 01:40:54.864968 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:40:54.864951 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe\": container with ID starting with 2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe not found: ID does not exist" containerID="2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe" Nov 04 01:40:54.865025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.864977 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe"} err="failed to get container status \"2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe\": rpc error: code = NotFound desc = could not find container \"2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe\": container with ID starting with 2461028ac0fb61dbc52e3e68a954a610bb1a72c2df277a34368b79b1d6c566fe not found: ID does not exist" Nov 04 01:40:54.865025 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.864992 4312 scope.go:122] "RemoveContainer" containerID="8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c" Nov 04 01:40:54.872741 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.872722 4312 scope.go:122] "RemoveContainer" containerID="8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c" Nov 04 01:40:54.873003 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:40:54.872982 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c\": container with ID starting with 8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c not found: ID does not exist" containerID="8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c" Nov 04 01:40:54.873053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.873015 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c"} err="failed to get container status \"8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c\": rpc error: code = NotFound desc = could not find container \"8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c\": container with ID starting with 8c69bd3ddbcb9350d97747a9af078c825663a77bd7bdbefc4dc3dfb8062f1b4c not found: ID does not exist" Nov 04 01:40:54.918412 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.918386 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dec1e7a-65d8-4940-94ac-d54178abc8b9-kube-api-access-8gh4c\" (UniqueName: \"kubernetes.io/projected/9dec1e7a-65d8-4940-94ac-d54178abc8b9-kube-api-access-8gh4c\") pod \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " Nov 04 01:40:54.918557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.918426 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17194cf9-23f2-4233-a166-2b7a73a00825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17194cf9-23f2-4233-a166-2b7a73a00825-plugins-config-volume\") pod \"17194cf9-23f2-4233-a166-2b7a73a00825\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " Nov 04 01:40:54.918557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.918463 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17194cf9-23f2-4233-a166-2b7a73a00825-kube-api-access-xhm9s\" (UniqueName: \"kubernetes.io/projected/17194cf9-23f2-4233-a166-2b7a73a00825-kube-api-access-xhm9s\") pod \"17194cf9-23f2-4233-a166-2b7a73a00825\" (UID: \"17194cf9-23f2-4233-a166-2b7a73a00825\") " Nov 04 01:40:54.918557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.918550 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dec1e7a-65d8-4940-94ac-d54178abc8b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dec1e7a-65d8-4940-94ac-d54178abc8b9-plugins-config-volume\") pod \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\" (UID: \"9dec1e7a-65d8-4940-94ac-d54178abc8b9\") " Nov 04 01:40:54.918836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.918809 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17194cf9-23f2-4233-a166-2b7a73a00825-plugins-config-volume" pod "17194cf9-23f2-4233-a166-2b7a73a00825" (UID: "17194cf9-23f2-4233-a166-2b7a73a00825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:54.918928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.918888 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dec1e7a-65d8-4940-94ac-d54178abc8b9-plugins-config-volume" pod "9dec1e7a-65d8-4940-94ac-d54178abc8b9" (UID: "9dec1e7a-65d8-4940-94ac-d54178abc8b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:54.920523 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.920504 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17194cf9-23f2-4233-a166-2b7a73a00825-kube-api-access-xhm9s" pod "17194cf9-23f2-4233-a166-2b7a73a00825" (UID: "17194cf9-23f2-4233-a166-2b7a73a00825"). InnerVolumeSpecName "kube-api-access-xhm9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:54.920666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:54.920618 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dec1e7a-65d8-4940-94ac-d54178abc8b9-kube-api-access-8gh4c" pod "9dec1e7a-65d8-4940-94ac-d54178abc8b9" (UID: "9dec1e7a-65d8-4940-94ac-d54178abc8b9"). InnerVolumeSpecName "kube-api-access-8gh4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:55.019555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.019479 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dec1e7a-65d8-4940-94ac-d54178abc8b9-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:55.019555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.019508 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gh4c\" (UniqueName: \"kubernetes.io/projected/9dec1e7a-65d8-4940-94ac-d54178abc8b9-kube-api-access-8gh4c\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:55.019555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.019518 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17194cf9-23f2-4233-a166-2b7a73a00825-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:55.019555 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.019527 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhm9s\" (UniqueName: \"kubernetes.io/projected/17194cf9-23f2-4233-a166-2b7a73a00825-kube-api-access-xhm9s\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:55.181647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.181620 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd"] Nov 04 01:40:55.185911 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.185888 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844tvhpd"] Nov 04 01:40:55.197138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.197117 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6"] Nov 04 01:40:55.203393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:55.203374 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468c6cfn6"] Nov 04 01:40:56.107793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:56.107758 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17194cf9-23f2-4233-a166-2b7a73a00825" path="/var/lib/kubelet/pods/17194cf9-23f2-4233-a166-2b7a73a00825/volumes" Nov 04 01:40:56.108179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:40:56.108117 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" path="/var/lib/kubelet/pods/9dec1e7a-65d8-4940-94ac-d54178abc8b9/volumes" Nov 04 01:40:59.459058 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:40:59.458960 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:41:02.046842 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.046787 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj"] Nov 04 01:41:02.049170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.049140 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c57jj"] Nov 04 01:41:02.073517 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.073480 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm"] Nov 04 01:41:02.073980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.073961 4312 cpu_manager.go:414] "containerMap: removing container" podUID="17194cf9-23f2-4233-a166-2b7a73a00825" containerName="payload-processing" Nov 04 01:41:02.074070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.073983 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17194cf9-23f2-4233-a166-2b7a73a00825" containerName="payload-processing" Nov 04 01:41:02.074070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074004 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f6aee319-7f4d-4d2c-b60c-a4240bb183e9" containerName="main" Nov 04 01:41:02.074070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074013 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6aee319-7f4d-4d2c-b60c-a4240bb183e9" containerName="main" Nov 04 01:41:02.074070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074030 4312 cpu_manager.go:414] "containerMap: removing container" podUID="901fe8b0-4ef4-4d90-898f-0834ee9b022a" containerName="maas-api" Nov 04 01:41:02.074070 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074038 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901fe8b0-4ef4-4d90-898f-0834ee9b022a" containerName="maas-api" Nov 04 01:41:02.074348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074077 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" containerName="payload-pre-processing" Nov 04 01:41:02.074348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074086 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" containerName="payload-pre-processing" Nov 04 01:41:02.074348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074201 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6aee319-7f4d-4d2c-b60c-a4240bb183e9" containerName="main" Nov 04 01:41:02.074348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074215 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="901fe8b0-4ef4-4d90-898f-0834ee9b022a" containerName="maas-api" Nov 04 01:41:02.074348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074230 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dec1e7a-65d8-4940-94ac-d54178abc8b9" containerName="payload-pre-processing" Nov 04 01:41:02.074348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.074242 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="17194cf9-23f2-4233-a166-2b7a73a00825" containerName="payload-processing" Nov 04 01:41:02.079152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.079131 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.086185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.086161 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rbkg2\"" Nov 04 01:41:02.086541 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.086514 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm"] Nov 04 01:41:02.090940 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.090921 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-042372-serving-cert\"" Nov 04 01:41:02.091226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.091211 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:41:02.091474 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.091454 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:41:02.092223 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.091956 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:41:02.107783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.107758 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded242f2-0636-4365-9431-1376f4ad6b37" path="/var/lib/kubelet/pods/ded242f2-0636-4365-9431-1376f4ad6b37/volumes" Nov 04 01:41:02.111578 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.111551 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2"] Nov 04 01:41:02.115706 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.115689 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.127537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.127505 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2"] Nov 04 01:41:02.134808 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.134784 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck"] Nov 04 01:41:02.138429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.138410 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.160733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.160704 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck"] Nov 04 01:41:02.177195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.177170 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8d98d1ab-dockercfg-9m42k\"" Nov 04 01:41:02.191378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.191355 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-042372\"" Nov 04 01:41:02.196471 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.196450 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8m6v\" (UniqueName: \"kubernetes.io/projected/88d29997-53bb-418c-b86a-79e9119e93cf-kube-api-access-q8m6v\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.196583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.196493 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.196750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.196730 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.196813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.196781 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d29997-53bb-418c-b86a-79e9119e93cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.298647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.298328 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhng\" (UniqueName: \"kubernetes.io/projected/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-kube-api-access-twhng\") pod \"payload-processing-e2e-worker-single-main-042372-745659844cmrck\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.298647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.298482 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dcdef-2143-4122-a45c-3001b3af85c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.299051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.298546 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.299187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.299087 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d29997-53bb-418c-b86a-79e9119e93cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.299187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.299140 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njhql\" (UniqueName: \"kubernetes.io/projected/538dcdef-2143-4122-a45c-3001b3af85c2-kube-api-access-njhql\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.299187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.299179 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8m6v\" (UniqueName: \"kubernetes.io/projected/88d29997-53bb-418c-b86a-79e9119e93cf-kube-api-access-q8m6v\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.299397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.299232 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.299397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.299261 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042372-745659844cmrck\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.301379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.301286 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d29997-53bb-418c-b86a-79e9119e93cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.304375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.303303 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.304375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.303401 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.314858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.313320 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8m6v\" (UniqueName: \"kubernetes.io/projected/88d29997-53bb-418c-b86a-79e9119e93cf-kube-api-access-q8m6v\") pod \"maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.397289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.397253 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:02.401048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.400642 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042372-745659844cmrck\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.401048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.400696 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhng\" (UniqueName: \"kubernetes.io/projected/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-kube-api-access-twhng\") pod \"payload-processing-e2e-worker-single-main-042372-745659844cmrck\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.401048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.400764 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dcdef-2143-4122-a45c-3001b3af85c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.401048 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.400822 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njhql\" (UniqueName: \"kubernetes.io/projected/538dcdef-2143-4122-a45c-3001b3af85c2-kube-api-access-njhql\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.401598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.401572 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042372-745659844cmrck\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.401751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.401728 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dcdef-2143-4122-a45c-3001b3af85c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.413327 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.413301 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njhql\" (UniqueName: \"kubernetes.io/projected/538dcdef-2143-4122-a45c-3001b3af85c2-kube-api-access-njhql\") pod \"payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.413677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.413652 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhng\" (UniqueName: \"kubernetes.io/projected/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-kube-api-access-twhng\") pod \"payload-processing-e2e-worker-single-main-042372-745659844cmrck\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.431343 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.431305 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.455945 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.455755 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.685294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.684716 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm"] Nov 04 01:41:02.685294 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:41:02.685225 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d29997_53bb_418c_b86a_79e9119e93cf.slice/crio-c6c533b864236702d2a2693747be81cb9cca3df4de032391961ad972b5ed2de0 WatchSource:0}: Error finding container c6c533b864236702d2a2693747be81cb9cca3df4de032391961ad972b5ed2de0: Status 404 returned error can't find the container with id c6c533b864236702d2a2693747be81cb9cca3df4de032391961ad972b5ed2de0 Nov 04 01:41:02.687192 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.687173 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:41:02.728236 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.728101 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2"] Nov 04 01:41:02.730961 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:41:02.730936 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538dcdef_2143_4122_a45c_3001b3af85c2.slice/crio-a84299f356c40f349c7d86fa11a18a3a634c67d15b88b69ad0e31e56f0264f5e WatchSource:0}: Error finding container a84299f356c40f349c7d86fa11a18a3a634c67d15b88b69ad0e31e56f0264f5e: Status 404 returned error can't find the container with id a84299f356c40f349c7d86fa11a18a3a634c67d15b88b69ad0e31e56f0264f5e Nov 04 01:41:02.768045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.768018 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck"] Nov 04 01:41:02.769450 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:41:02.769421 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245f28b1_c0c4_44dc_aa6a_fe3ce7c80148.slice/crio-212f32cb49502fb5b9b20575457c40b1807f8136eb324abcce2dfb3005c314b2 WatchSource:0}: Error finding container 212f32cb49502fb5b9b20575457c40b1807f8136eb324abcce2dfb3005c314b2: Status 404 returned error can't find the container with id 212f32cb49502fb5b9b20575457c40b1807f8136eb324abcce2dfb3005c314b2 Nov 04 01:41:02.902971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.902933 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" event={"ID":"88d29997-53bb-418c-b86a-79e9119e93cf","Type":"ContainerStarted","Data":"c6c533b864236702d2a2693747be81cb9cca3df4de032391961ad972b5ed2de0"} Nov 04 01:41:02.904528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.904494 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" event={"ID":"538dcdef-2143-4122-a45c-3001b3af85c2","Type":"ContainerStarted","Data":"f6047f3292b5226f4243482990d607f225e76fbdea79b70491e443b5db76a9c0"} Nov 04 01:41:02.904676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.904550 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" event={"ID":"538dcdef-2143-4122-a45c-3001b3af85c2","Type":"ContainerStarted","Data":"a84299f356c40f349c7d86fa11a18a3a634c67d15b88b69ad0e31e56f0264f5e"} Nov 04 01:41:02.904752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.904732 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:02.906153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.906130 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" event={"ID":"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148","Type":"ContainerStarted","Data":"9e9188359d997b6dff5b358ff1a40d2b4d30b22bbf4d4d63561f661ef19907ba"} Nov 04 01:41:02.906261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.906166 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" event={"ID":"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148","Type":"ContainerStarted","Data":"212f32cb49502fb5b9b20575457c40b1807f8136eb324abcce2dfb3005c314b2"} Nov 04 01:41:02.906261 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.906203 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:02.924429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:02.924378 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" podStartSLOduration=0.924361775 podStartE2EDuration="924.361775ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:02.92227185 +0000 UTC m=+2294.945203871" watchObservedRunningTime="2026-11-04 01:41:02.924361775 +0000 UTC m=+2294.947293783" Nov 04 01:41:03.913564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:03.913524 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" event={"ID":"88d29997-53bb-418c-b86a-79e9119e93cf","Type":"ContainerStarted","Data":"7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7"} Nov 04 01:41:03.934987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:03.934923 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" podStartSLOduration=1.387651622 podStartE2EDuration="1.934906036s" totalImagesPullingTime="547.254414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:03.934613742 +0000 UTC m=+2295.957545750" watchObservedRunningTime="2026-11-04 01:41:03.934906036 +0000 UTC m=+2295.957838043" Nov 04 01:41:03.936740 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:03.936698 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" podStartSLOduration=1.936688195 podStartE2EDuration="1.936688195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:02.947294225 +0000 UTC m=+2294.970226232" watchObservedRunningTime="2026-11-04 01:41:03.936688195 +0000 UTC m=+2295.959620244" Nov 04 01:41:04.918862 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:04.918833 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:10.931750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:10.931723 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:13.027825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:13.027793 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv"] Nov 04 01:41:13.031909 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:13.031885 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8jkxv"] Nov 04 01:41:13.916601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:13.916572 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:13.916775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:13.916637 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:14.107102 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:14.107064 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d53e28-5a20-486c-9692-6437be49d41c" path="/var/lib/kubelet/pods/16d53e28-5a20-486c-9692-6437be49d41c/volumes" Nov 04 01:41:21.861733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:21.861700 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck"] Nov 04 01:41:21.879146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:21.879104 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2"] Nov 04 01:41:22.616767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:22.616734 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm"] Nov 04 01:41:22.616995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:22.616968 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" podUID="88d29997-53bb-418c-b86a-79e9119e93cf" containerName="maas-api" containerID="cri-o://7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7" gracePeriod=30 Nov 04 01:41:22.872313 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:22.872250 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:23.011758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.011725 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d29997-53bb-418c-b86a-79e9119e93cf-kube-api-access-q8m6v\" (UniqueName: \"kubernetes.io/projected/88d29997-53bb-418c-b86a-79e9119e93cf-kube-api-access-q8m6v\") pod \"88d29997-53bb-418c-b86a-79e9119e93cf\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " Nov 04 01:41:23.011941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.011765 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-maas-api-tls\") pod \"88d29997-53bb-418c-b86a-79e9119e93cf\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " Nov 04 01:41:23.011941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.011827 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d29997-53bb-418c-b86a-79e9119e93cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d29997-53bb-418c-b86a-79e9119e93cf-openshift-service-ca\") pod \"88d29997-53bb-418c-b86a-79e9119e93cf\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " Nov 04 01:41:23.011941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.011861 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-metrics-certs\") pod \"88d29997-53bb-418c-b86a-79e9119e93cf\" (UID: \"88d29997-53bb-418c-b86a-79e9119e93cf\") " Nov 04 01:41:23.012253 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.012222 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d29997-53bb-418c-b86a-79e9119e93cf-openshift-service-ca" pod "88d29997-53bb-418c-b86a-79e9119e93cf" (UID: "88d29997-53bb-418c-b86a-79e9119e93cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:23.013883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.013856 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-maas-api-tls" pod "88d29997-53bb-418c-b86a-79e9119e93cf" (UID: "88d29997-53bb-418c-b86a-79e9119e93cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:41:23.013991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.013885 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-metrics-certs" pod "88d29997-53bb-418c-b86a-79e9119e93cf" (UID: "88d29997-53bb-418c-b86a-79e9119e93cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:41:23.013991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.013970 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d29997-53bb-418c-b86a-79e9119e93cf-kube-api-access-q8m6v" pod "88d29997-53bb-418c-b86a-79e9119e93cf" (UID: "88d29997-53bb-418c-b86a-79e9119e93cf"). InnerVolumeSpecName "kube-api-access-q8m6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:23.024492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.024466 4312 generic.go:356] "Generic (PLEG): container finished" podID="88d29997-53bb-418c-b86a-79e9119e93cf" containerID="7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7" exitCode=0 Nov 04 01:41:23.024596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.024512 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" Nov 04 01:41:23.024596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.024546 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" event={"ID":"88d29997-53bb-418c-b86a-79e9119e93cf","Type":"ContainerDied","Data":"7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7"} Nov 04 01:41:23.024691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.024608 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm" event={"ID":"88d29997-53bb-418c-b86a-79e9119e93cf","Type":"ContainerDied","Data":"c6c533b864236702d2a2693747be81cb9cca3df4de032391961ad972b5ed2de0"} Nov 04 01:41:23.024691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.024630 4312 scope.go:122] "RemoveContainer" containerID="7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7" Nov 04 01:41:23.035118 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.035101 4312 scope.go:122] "RemoveContainer" containerID="7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7" Nov 04 01:41:23.035408 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:41:23.035381 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7\": container with ID starting with 7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7 not found: ID does not exist" containerID="7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7" Nov 04 01:41:23.035505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.035420 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7"} err="failed to get container status \"7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7\": rpc error: code = NotFound desc = could not find container \"7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7\": container with ID starting with 7dd0b49940527b849eebe7667a2201446860ed5ea437a71a9fee7e13e7d796e7 not found: ID does not exist" Nov 04 01:41:23.048071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.048044 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm"] Nov 04 01:41:23.050303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.050283 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042372-5ccb75984b-zkcrm"] Nov 04 01:41:23.112623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.112597 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8m6v\" (UniqueName: \"kubernetes.io/projected/88d29997-53bb-418c-b86a-79e9119e93cf-kube-api-access-q8m6v\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:23.112623 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.112624 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:23.112798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.112639 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d29997-53bb-418c-b86a-79e9119e93cf-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:23.112798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.112653 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d29997-53bb-418c-b86a-79e9119e93cf-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:23.243764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.243667 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c"] Nov 04 01:41:23.243950 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:23.243920 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" containerID="cri-o://cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb" gracePeriod=30 Nov 04 01:41:24.107925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.107892 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d29997-53bb-418c-b86a-79e9119e93cf" path="/var/lib/kubelet/pods/88d29997-53bb-418c-b86a-79e9119e93cf/volumes" Nov 04 01:41:24.548189 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.548150 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj"] Nov 04 01:41:24.559533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.554605 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s4cj"] Nov 04 01:41:24.627586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.627546 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594"] Nov 04 01:41:24.628079 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.628059 4312 cpu_manager.go:414] "containerMap: removing container" podUID="88d29997-53bb-418c-b86a-79e9119e93cf" containerName="maas-api" Nov 04 01:41:24.628184 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.628080 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d29997-53bb-418c-b86a-79e9119e93cf" containerName="maas-api" Nov 04 01:41:24.628245 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.628230 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d29997-53bb-418c-b86a-79e9119e93cf" containerName="maas-api" Nov 04 01:41:24.632958 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.632939 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.652207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.652188 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:41:24.657498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.657476 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:41:24.659847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.659135 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:41:24.659847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.659169 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:41:24.660880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.660845 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594"] Nov 04 01:41:24.729389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.729326 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfsj\" (UniqueName: \"kubernetes.io/projected/b9985080-b9d5-4c7a-925d-24982dcb4f0e-kube-api-access-hgfsj\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.729573 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.729451 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.729647 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.729583 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.831164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.831082 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.831164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.831156 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfsj\" (UniqueName: \"kubernetes.io/projected/b9985080-b9d5-4c7a-925d-24982dcb4f0e-kube-api-access-hgfsj\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.831385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.831211 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.831946 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.831917 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.833562 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.833541 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.843407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.843328 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfsj\" (UniqueName: \"kubernetes.io/projected/b9985080-b9d5-4c7a-925d-24982dcb4f0e-kube-api-access-hgfsj\") pod \"maas-api-revoke-sub-3d4156f1-gz594\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:24.945425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:24.945391 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:25.222965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:25.222941 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594"] Nov 04 01:41:25.225173 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:41:25.225143 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9985080_b9d5_4c7a_925d_24982dcb4f0e.slice/crio-33e692ec382e891b4ce38346975af2d233b3f9250cc2c864164c88b9f98c99bb WatchSource:0}: Error finding container 33e692ec382e891b4ce38346975af2d233b3f9250cc2c864164c88b9f98c99bb: Status 404 returned error can't find the container with id 33e692ec382e891b4ce38346975af2d233b3f9250cc2c864164c88b9f98c99bb Nov 04 01:41:26.045037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:26.045001 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" event={"ID":"b9985080-b9d5-4c7a-925d-24982dcb4f0e","Type":"ContainerStarted","Data":"3bff1dac4952e35a8a0821c3258e45513821368ccb678c4b4b001a43575cc615"} Nov 04 01:41:26.045203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:26.045057 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" event={"ID":"b9985080-b9d5-4c7a-925d-24982dcb4f0e","Type":"ContainerStarted","Data":"33e692ec382e891b4ce38346975af2d233b3f9250cc2c864164c88b9f98c99bb"} Nov 04 01:41:26.106775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:26.106746 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab27f145-2165-428f-a3b3-512c70ff2320" path="/var/lib/kubelet/pods/ab27f145-2165-428f-a3b3-512c70ff2320/volumes" Nov 04 01:41:26.862528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:26.862470 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" podUID="245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" containerName="payload-processing" containerID="cri-o://9e9188359d997b6dff5b358ff1a40d2b4d30b22bbf4d4d63561f661ef19907ba" gracePeriod=25 Nov 04 01:41:26.879991 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:26.879942 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" podUID="538dcdef-2143-4122-a45c-3001b3af85c2" containerName="payload-pre-processing" containerID="cri-o://f6047f3292b5226f4243482990d607f225e76fbdea79b70491e443b5db76a9c0" gracePeriod=25 Nov 04 01:41:27.056140 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.056104 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" event={"ID":"b9985080-b9d5-4c7a-925d-24982dcb4f0e","Type":"ContainerDied","Data":"3bff1dac4952e35a8a0821c3258e45513821368ccb678c4b4b001a43575cc615"} Nov 04 01:41:27.056342 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.056308 4312 generic.go:356] "Generic (PLEG): container finished" podID="b9985080-b9d5-4c7a-925d-24982dcb4f0e" containerID="3bff1dac4952e35a8a0821c3258e45513821368ccb678c4b4b001a43575cc615" exitCode=0 Nov 04 01:41:27.057906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.057742 4312 generic.go:356] "Generic (PLEG): container finished" podID="538dcdef-2143-4122-a45c-3001b3af85c2" containerID="f6047f3292b5226f4243482990d607f225e76fbdea79b70491e443b5db76a9c0" exitCode=0 Nov 04 01:41:27.057906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.057849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" event={"ID":"538dcdef-2143-4122-a45c-3001b3af85c2","Type":"ContainerDied","Data":"f6047f3292b5226f4243482990d607f225e76fbdea79b70491e443b5db76a9c0"} Nov 04 01:41:27.059708 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.059670 4312 generic.go:356] "Generic (PLEG): container finished" podID="245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" containerID="9e9188359d997b6dff5b358ff1a40d2b4d30b22bbf4d4d63561f661ef19907ba" exitCode=0 Nov 04 01:41:27.059826 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.059729 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" event={"ID":"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148","Type":"ContainerDied","Data":"9e9188359d997b6dff5b358ff1a40d2b4d30b22bbf4d4d63561f661ef19907ba"} Nov 04 01:41:27.146569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.146546 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:27.159957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.159931 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:27.255674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.255643 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-plugins-config-volume\") pod \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " Nov 04 01:41:27.255836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.255680 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-kube-api-access-twhng\" (UniqueName: \"kubernetes.io/projected/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-kube-api-access-twhng\") pod \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\" (UID: \"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148\") " Nov 04 01:41:27.255836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.255727 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538dcdef-2143-4122-a45c-3001b3af85c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dcdef-2143-4122-a45c-3001b3af85c2-plugins-config-volume\") pod \"538dcdef-2143-4122-a45c-3001b3af85c2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " Nov 04 01:41:27.255836 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.255806 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538dcdef-2143-4122-a45c-3001b3af85c2-kube-api-access-njhql\" (UniqueName: \"kubernetes.io/projected/538dcdef-2143-4122-a45c-3001b3af85c2-kube-api-access-njhql\") pod \"538dcdef-2143-4122-a45c-3001b3af85c2\" (UID: \"538dcdef-2143-4122-a45c-3001b3af85c2\") " Nov 04 01:41:27.256130 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.256101 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-plugins-config-volume" pod "245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" (UID: "245f28b1-c0c4-44dc-aa6a-fe3ce7c80148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:27.256242 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.256158 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538dcdef-2143-4122-a45c-3001b3af85c2-plugins-config-volume" pod "538dcdef-2143-4122-a45c-3001b3af85c2" (UID: "538dcdef-2143-4122-a45c-3001b3af85c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:27.257796 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.257768 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-kube-api-access-twhng" pod "245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" (UID: "245f28b1-c0c4-44dc-aa6a-fe3ce7c80148"). InnerVolumeSpecName "kube-api-access-twhng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:27.257900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.257825 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538dcdef-2143-4122-a45c-3001b3af85c2-kube-api-access-njhql" pod "538dcdef-2143-4122-a45c-3001b3af85c2" (UID: "538dcdef-2143-4122-a45c-3001b3af85c2"). InnerVolumeSpecName "kube-api-access-njhql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:27.357469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.357441 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:27.357469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.357465 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhng\" (UniqueName: \"kubernetes.io/projected/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148-kube-api-access-twhng\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:27.357469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.357474 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dcdef-2143-4122-a45c-3001b3af85c2-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:27.357659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:27.357484 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njhql\" (UniqueName: \"kubernetes.io/projected/538dcdef-2143-4122-a45c-3001b3af85c2-kube-api-access-njhql\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.066077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.066045 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" Nov 04 01:41:28.066077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.066073 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2" event={"ID":"538dcdef-2143-4122-a45c-3001b3af85c2","Type":"ContainerDied","Data":"a84299f356c40f349c7d86fa11a18a3a634c67d15b88b69ad0e31e56f0264f5e"} Nov 04 01:41:28.066594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.066133 4312 scope.go:122] "RemoveContainer" containerID="f6047f3292b5226f4243482990d607f225e76fbdea79b70491e443b5db76a9c0" Nov 04 01:41:28.067632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.067606 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" Nov 04 01:41:28.067758 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.067635 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck" event={"ID":"245f28b1-c0c4-44dc-aa6a-fe3ce7c80148","Type":"ContainerDied","Data":"212f32cb49502fb5b9b20575457c40b1807f8136eb324abcce2dfb3005c314b2"} Nov 04 01:41:28.078296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.078277 4312 scope.go:122] "RemoveContainer" containerID="9e9188359d997b6dff5b358ff1a40d2b4d30b22bbf4d4d63561f661ef19907ba" Nov 04 01:41:28.096135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.096104 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck"] Nov 04 01:41:28.101701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.101673 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042372-745659844cmrck"] Nov 04 01:41:28.110129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.110106 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" path="/var/lib/kubelet/pods/245f28b1-c0c4-44dc-aa6a-fe3ce7c80148/volumes" Nov 04 01:41:28.116849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.116826 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2"] Nov 04 01:41:28.124755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.124732 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042372-6468cpmxn2"] Nov 04 01:41:28.207294 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.207270 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:28.367131 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.367107 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9985080-b9d5-4c7a-925d-24982dcb4f0e-kube-api-access-hgfsj\" (UniqueName: \"kubernetes.io/projected/b9985080-b9d5-4c7a-925d-24982dcb4f0e-kube-api-access-hgfsj\") pod \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " Nov 04 01:41:28.367274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.367154 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-xks-ca\") pod \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " Nov 04 01:41:28.367389 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.367320 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-openshift-ca\") pod \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\" (UID: \"b9985080-b9d5-4c7a-925d-24982dcb4f0e\") " Nov 04 01:41:28.367767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.367731 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-openshift-ca" pod "b9985080-b9d5-4c7a-925d-24982dcb4f0e" (UID: "b9985080-b9d5-4c7a-925d-24982dcb4f0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:28.369081 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.369060 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9985080-b9d5-4c7a-925d-24982dcb4f0e-kube-api-access-hgfsj" pod "b9985080-b9d5-4c7a-925d-24982dcb4f0e" (UID: "b9985080-b9d5-4c7a-925d-24982dcb4f0e"). InnerVolumeSpecName "kube-api-access-hgfsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:28.369311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.369297 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-xks-ca" pod "b9985080-b9d5-4c7a-925d-24982dcb4f0e" (UID: "b9985080-b9d5-4c7a-925d-24982dcb4f0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:41:28.468226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.468189 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfsj\" (UniqueName: \"kubernetes.io/projected/b9985080-b9d5-4c7a-925d-24982dcb4f0e-kube-api-access-hgfsj\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.468226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.468220 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.468437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.468234 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9985080-b9d5-4c7a-925d-24982dcb4f0e-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.483180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.483157 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:41:28.568699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.568664 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-credential-socket\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.568880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.568706 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-data\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.568880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.568741 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-istio-token\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-istio-token\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.568880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.568795 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-certs\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.568880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.568841 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-socket\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.569120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.568994 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-credential-socket" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:28.569120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569014 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-kube-api-access-zkfg9\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-kube-api-access-zkfg9\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.569120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569081 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-envoy\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.569120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569004 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-data" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:28.569318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569122 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61130f1c-058d-459a-a016-08a54774ca00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61130f1c-058d-459a-a016-08a54774ca00-istiod-ca-cert\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.569318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569126 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-socket" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:28.569318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569166 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-certs" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:28.569318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569175 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61130f1c-058d-459a-a016-08a54774ca00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61130f1c-058d-459a-a016-08a54774ca00-istio-podinfo\") pod \"61130f1c-058d-459a-a016-08a54774ca00\" (UID: \"61130f1c-058d-459a-a016-08a54774ca00\") " Nov 04 01:41:28.569537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569521 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61130f1c-058d-459a-a016-08a54774ca00-istiod-ca-cert" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:28.569677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569654 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61130f1c-058d-459a-a016-08a54774ca00-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.569734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569684 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.569734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569699 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.569734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569713 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.569734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.569726 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.571159 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.571138 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-envoy" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:28.571228 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.571154 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61130f1c-058d-459a-a016-08a54774ca00-istio-podinfo" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:41:28.571480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.571464 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-istio-token" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:28.571521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.571480 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-kube-api-access-zkfg9" pod "61130f1c-058d-459a-a016-08a54774ca00" (UID: "61130f1c-058d-459a-a016-08a54774ca00"). InnerVolumeSpecName "kube-api-access-zkfg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:28.670600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.670526 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfg9\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-kube-api-access-zkfg9\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.670600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.670553 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61130f1c-058d-459a-a016-08a54774ca00-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.670600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.670563 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61130f1c-058d-459a-a016-08a54774ca00-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:28.670600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:28.670572 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61130f1c-058d-459a-a016-08a54774ca00-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:29.075271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.075239 4312 generic.go:356] "Generic (PLEG): container finished" podID="61130f1c-058d-459a-a016-08a54774ca00" containerID="cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb" exitCode=0 Nov 04 01:41:29.075735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.075308 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" Nov 04 01:41:29.075735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.075322 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" event={"ID":"61130f1c-058d-459a-a016-08a54774ca00","Type":"ContainerDied","Data":"cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb"} Nov 04 01:41:29.075735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.075438 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c" event={"ID":"61130f1c-058d-459a-a016-08a54774ca00","Type":"ContainerDied","Data":"edd0dfe084343a1c06e49c53e66bd38cc7626661d7f88c309d283e6e45698ec8"} Nov 04 01:41:29.075735 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.075467 4312 scope.go:122] "RemoveContainer" containerID="cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb" Nov 04 01:41:29.076832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.076816 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" Nov 04 01:41:29.076925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.076831 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594" event={"ID":"b9985080-b9d5-4c7a-925d-24982dcb4f0e","Type":"ContainerDied","Data":"33e692ec382e891b4ce38346975af2d233b3f9250cc2c864164c88b9f98c99bb"} Nov 04 01:41:29.076925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.076876 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33e692ec382e891b4ce38346975af2d233b3f9250cc2c864164c88b9f98c99bb" Nov 04 01:41:29.087376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.087356 4312 scope.go:122] "RemoveContainer" containerID="3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b" Nov 04 01:41:29.096181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.096156 4312 scope.go:122] "RemoveContainer" containerID="cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb" Nov 04 01:41:29.096485 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:41:29.096465 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb\": container with ID starting with cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb not found: ID does not exist" containerID="cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb" Nov 04 01:41:29.096570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.096493 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb"} err="failed to get container status \"cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb\": rpc error: code = NotFound desc = could not find container \"cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb\": container with ID starting with cc090639fba58549b32ae2718c94f091447681b2296657dcd0c4a6503ac409bb not found: ID does not exist" Nov 04 01:41:29.096570 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.096510 4312 scope.go:122] "RemoveContainer" containerID="3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b" Nov 04 01:41:29.096752 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:41:29.096736 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b\": container with ID starting with 3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b not found: ID does not exist" containerID="3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b" Nov 04 01:41:29.096813 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.096756 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b"} err="failed to get container status \"3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b\": rpc error: code = NotFound desc = could not find container \"3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b\": container with ID starting with 3e5bdd754e053322ec548f5f83be93d0e9b7916c7171c243b0c20073ccd4805b not found: ID does not exist" Nov 04 01:41:29.103450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.103428 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c"] Nov 04 01:41:29.111827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:29.111802 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042372-openshift-default-6f8764874f7rt4c"] Nov 04 01:41:29.456023 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:41:29.455937 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:41:30.107642 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:30.107597 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538dcdef-2143-4122-a45c-3001b3af85c2" path="/var/lib/kubelet/pods/538dcdef-2143-4122-a45c-3001b3af85c2/volumes" Nov 04 01:41:30.108323 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:30.108301 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61130f1c-058d-459a-a016-08a54774ca00" path="/var/lib/kubelet/pods/61130f1c-058d-459a-a016-08a54774ca00/volumes" Nov 04 01:41:37.043359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:37.042398 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg"] Nov 04 01:41:37.045889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:37.045861 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xvccg"] Nov 04 01:41:38.121817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:38.121779 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd272834-20cc-406c-8644-4af71337fc2c" path="/var/lib/kubelet/pods/cd272834-20cc-406c-8644-4af71337fc2c/volumes" Nov 04 01:41:47.040839 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:47.040779 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc"] Nov 04 01:41:47.043667 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:47.043640 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fdwbc"] Nov 04 01:41:48.109114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:48.109074 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c4dd25-38bf-4580-b4e4-877d023d4044" path="/var/lib/kubelet/pods/16c4dd25-38bf-4580-b4e4-877d023d4044/volumes" Nov 04 01:41:50.649747 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:41:50.649712 4312 scope.go:122] "RemoveContainer" containerID="4b4347a0b5f5fed55eb9578afc5bc7a0d12a28616664f3e85ea9c4d397857b3a" Nov 04 01:41:59.456944 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:41:59.456906 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:42:08.024511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:08.024471 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2"] Nov 04 01:42:08.027002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:08.026980 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jxmr2"] Nov 04 01:42:08.107470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:08.107437 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff67d663-0c56-4b03-a1a3-db946bab457a" path="/var/lib/kubelet/pods/ff67d663-0c56-4b03-a1a3-db946bab457a/volumes" Nov 04 01:42:25.026024 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:25.025986 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb"] Nov 04 01:42:25.028404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:25.028380 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6rgb"] Nov 04 01:42:26.106410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:26.106380 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7532eea5-d5f2-45e1-9d75-cb09717aeba5" path="/var/lib/kubelet/pods/7532eea5-d5f2-45e1-9d75-cb09717aeba5/volumes" Nov 04 01:42:29.457387 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:42:29.457301 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:42:41.024283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.024239 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8"] Nov 04 01:42:41.026865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.026841 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fz7d8"] Nov 04 01:42:41.164890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.164855 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq"] Nov 04 01:42:41.165299 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165285 4312 cpu_manager.go:414] "containerMap: removing container" podUID="245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" containerName="payload-processing" Nov 04 01:42:41.165360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165300 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" containerName="payload-processing" Nov 04 01:42:41.165360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165319 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b9985080-b9d5-4c7a-925d-24982dcb4f0e" containerName="revoke-keys" Nov 04 01:42:41.165360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165324 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9985080-b9d5-4c7a-925d-24982dcb4f0e" containerName="revoke-keys" Nov 04 01:42:41.165360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165352 4312 cpu_manager.go:414] "containerMap: removing container" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" Nov 04 01:42:41.165360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165360 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165370 4312 cpu_manager.go:414] "containerMap: removing container" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165375 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165383 4312 cpu_manager.go:414] "containerMap: removing container" podUID="538dcdef-2143-4122-a45c-3001b3af85c2" containerName="payload-pre-processing" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165387 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538dcdef-2143-4122-a45c-3001b3af85c2" containerName="payload-pre-processing" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165454 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="245f28b1-c0c4-44dc-aa6a-fe3ce7c80148" containerName="payload-processing" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165464 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9985080-b9d5-4c7a-925d-24982dcb4f0e" containerName="revoke-keys" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165472 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165478 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="61130f1c-058d-459a-a016-08a54774ca00" containerName="istio-proxy" Nov 04 01:42:41.165531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.165484 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="538dcdef-2143-4122-a45c-3001b3af85c2" containerName="payload-pre-processing" Nov 04 01:42:41.168834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.168818 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.175810 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.175788 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ss6z5\"" Nov 04 01:42:41.191047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.190981 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq"] Nov 04 01:42:41.276747 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.276671 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgkw\" (UniqueName: \"kubernetes.io/projected/40914eb2-0c22-498c-b294-04f0fc9daf6c-kube-api-access-7jgkw\") pod \"kuadrant-operator-controller-manager-678d87766f-246jq\" (UID: \"40914eb2-0c22-498c-b294-04f0fc9daf6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.276747 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.276719 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40914eb2-0c22-498c-b294-04f0fc9daf6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-246jq\" (UID: \"40914eb2-0c22-498c-b294-04f0fc9daf6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.377936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.377905 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgkw\" (UniqueName: \"kubernetes.io/projected/40914eb2-0c22-498c-b294-04f0fc9daf6c-kube-api-access-7jgkw\") pod \"kuadrant-operator-controller-manager-678d87766f-246jq\" (UID: \"40914eb2-0c22-498c-b294-04f0fc9daf6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.378093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.377951 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40914eb2-0c22-498c-b294-04f0fc9daf6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-246jq\" (UID: \"40914eb2-0c22-498c-b294-04f0fc9daf6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.378324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.378308 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40914eb2-0c22-498c-b294-04f0fc9daf6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-246jq\" (UID: \"40914eb2-0c22-498c-b294-04f0fc9daf6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.390218 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.390195 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgkw\" (UniqueName: \"kubernetes.io/projected/40914eb2-0c22-498c-b294-04f0fc9daf6c-kube-api-access-7jgkw\") pod \"kuadrant-operator-controller-manager-678d87766f-246jq\" (UID: \"40914eb2-0c22-498c-b294-04f0fc9daf6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.481949 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.481921 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:41.767602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:41.767570 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq"] Nov 04 01:42:41.771210 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:42:41.771181 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40914eb2_0c22_498c_b294_04f0fc9daf6c.slice/crio-ad887937d2facaeebf66eb4d16b68811d025af70043c5a644cb26927907f393d WatchSource:0}: Error finding container ad887937d2facaeebf66eb4d16b68811d025af70043c5a644cb26927907f393d: Status 404 returned error can't find the container with id ad887937d2facaeebf66eb4d16b68811d025af70043c5a644cb26927907f393d Nov 04 01:42:42.107099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:42.107065 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aaa7aff-0210-4ed9-bb4c-5b796375d1d2" path="/var/lib/kubelet/pods/2aaa7aff-0210-4ed9-bb4c-5b796375d1d2/volumes" Nov 04 01:42:42.651020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:42.650985 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" event={"ID":"40914eb2-0c22-498c-b294-04f0fc9daf6c","Type":"ContainerStarted","Data":"ad887937d2facaeebf66eb4d16b68811d025af70043c5a644cb26927907f393d"} Nov 04 01:42:46.686936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:46.686854 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" event={"ID":"40914eb2-0c22-498c-b294-04f0fc9daf6c","Type":"ContainerStarted","Data":"664bde27f2a8c0f2add24a4a752002ddf27ed388177d647db274a5b196bd8bcc"} Nov 04 01:42:46.687619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:46.687006 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:46.712004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:46.711949 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" podStartSLOduration=1.108826773 podStartE2EDuration="5.711931836s" totalImagesPullingTime="4.603105063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:46.709490386 +0000 UTC m=+2398.732422421" watchObservedRunningTime="2026-11-04 01:42:46.711931836 +0000 UTC m=+2398.734863844" Nov 04 01:42:50.701534 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:50.701502 4312 scope.go:122] "RemoveContainer" containerID="d0a41812665904138afc968293ebbd6b074e14cb4cf76d3dfa7987bb2730fa9e" Nov 04 01:42:50.710432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:50.710409 4312 scope.go:122] "RemoveContainer" containerID="a46bbccac35b8fee833d832807fdd666e8d24ab69d1c5ddd32df481991f8f389" Nov 04 01:42:50.718383 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:50.718366 4312 scope.go:122] "RemoveContainer" containerID="014ae7e1328aaa60bdadb622f85de9b8941268f4e5fa2ecd482731be032e48fa" Nov 04 01:42:50.725861 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:50.725843 4312 scope.go:122] "RemoveContainer" containerID="e29582694b1aa07ed77d39a2ccbb11d04069bbaadcb738bba381cd5875f526e3" Nov 04 01:42:55.027494 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:55.027455 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92"] Nov 04 01:42:55.030633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:55.030600 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-5gw92"] Nov 04 01:42:56.107690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:56.107652 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf43bbb-c925-42c4-aff8-fa7cc5616087" path="/var/lib/kubelet/pods/0bf43bbb-c925-42c4-aff8-fa7cc5616087/volumes" Nov 04 01:42:57.695524 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:42:57.695491 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-246jq" Nov 04 01:42:59.457606 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:42:59.457569 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:43:09.235437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.235405 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q"] Nov 04 01:43:09.240729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.240698 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.247271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.247249 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:43:09.247490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.247302 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:43:09.247490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.247484 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:43:09.247680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.247419 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:43:09.250731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.250702 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q"] Nov 04 01:43:09.305687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.305651 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb72p\" (UniqueName: \"kubernetes.io/projected/30eb8587-68bf-46ee-90a9-d55b3aa61116-kube-api-access-gb72p\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.305855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.305713 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.305855 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.305811 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.407361 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.407308 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.407554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.407388 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.407554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.407454 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb72p\" (UniqueName: \"kubernetes.io/projected/30eb8587-68bf-46ee-90a9-d55b3aa61116-kube-api-access-gb72p\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.407959 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.407936 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.409725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.409701 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.419953 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.419928 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb72p\" (UniqueName: \"kubernetes.io/projected/30eb8587-68bf-46ee-90a9-d55b3aa61116-kube-api-access-gb72p\") pod \"maas-api-revoke-sub-d576e68b-rdt7q\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.555756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.555701 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:09.840385 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:09.840357 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q"] Nov 04 01:43:09.842106 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:43:09.842078 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30eb8587_68bf_46ee_90a9_d55b3aa61116.slice/crio-c4b412496616cc9baa399d1e844bf3b413cf8240ecb480eb91013047a693304b WatchSource:0}: Error finding container c4b412496616cc9baa399d1e844bf3b413cf8240ecb480eb91013047a693304b: Status 404 returned error can't find the container with id c4b412496616cc9baa399d1e844bf3b413cf8240ecb480eb91013047a693304b Nov 04 01:43:10.845002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:10.844969 4312 generic.go:356] "Generic (PLEG): container finished" podID="30eb8587-68bf-46ee-90a9-d55b3aa61116" containerID="d5201de6a9f5ea28089d9023e1096560b32e4e2b04cf84ec98e3a1aafffd065e" exitCode=0 Nov 04 01:43:10.845432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:10.845014 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" event={"ID":"30eb8587-68bf-46ee-90a9-d55b3aa61116","Type":"ContainerDied","Data":"d5201de6a9f5ea28089d9023e1096560b32e4e2b04cf84ec98e3a1aafffd065e"} Nov 04 01:43:10.845432 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:10.845053 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" event={"ID":"30eb8587-68bf-46ee-90a9-d55b3aa61116","Type":"ContainerStarted","Data":"c4b412496616cc9baa399d1e844bf3b413cf8240ecb480eb91013047a693304b"} Nov 04 01:43:11.987446 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:11.987423 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:12.031475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.031442 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-xks-ca\") pod \"30eb8587-68bf-46ee-90a9-d55b3aa61116\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " Nov 04 01:43:12.031666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.031514 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30eb8587-68bf-46ee-90a9-d55b3aa61116-kube-api-access-gb72p\" (UniqueName: \"kubernetes.io/projected/30eb8587-68bf-46ee-90a9-d55b3aa61116-kube-api-access-gb72p\") pod \"30eb8587-68bf-46ee-90a9-d55b3aa61116\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " Nov 04 01:43:12.031666 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.031601 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-openshift-ca\") pod \"30eb8587-68bf-46ee-90a9-d55b3aa61116\" (UID: \"30eb8587-68bf-46ee-90a9-d55b3aa61116\") " Nov 04 01:43:12.032077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.032047 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-openshift-ca" pod "30eb8587-68bf-46ee-90a9-d55b3aa61116" (UID: "30eb8587-68bf-46ee-90a9-d55b3aa61116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:12.033551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.033527 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-xks-ca" pod "30eb8587-68bf-46ee-90a9-d55b3aa61116" (UID: "30eb8587-68bf-46ee-90a9-d55b3aa61116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:43:12.033716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.033690 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30eb8587-68bf-46ee-90a9-d55b3aa61116-kube-api-access-gb72p" pod "30eb8587-68bf-46ee-90a9-d55b3aa61116" (UID: "30eb8587-68bf-46ee-90a9-d55b3aa61116"). InnerVolumeSpecName "kube-api-access-gb72p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:12.132619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.132589 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:12.132619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.132615 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb8587-68bf-46ee-90a9-d55b3aa61116-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:12.132793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.132624 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb72p\" (UniqueName: \"kubernetes.io/projected/30eb8587-68bf-46ee-90a9-d55b3aa61116-kube-api-access-gb72p\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:12.856150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.856114 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" event={"ID":"30eb8587-68bf-46ee-90a9-d55b3aa61116","Type":"ContainerDied","Data":"c4b412496616cc9baa399d1e844bf3b413cf8240ecb480eb91013047a693304b"} Nov 04 01:43:12.856374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.856168 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4b412496616cc9baa399d1e844bf3b413cf8240ecb480eb91013047a693304b" Nov 04 01:43:12.856374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:12.856127 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q" Nov 04 01:43:25.528752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:25.528717 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:43:25.529547 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:25.529514 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-m8lgv" podUID="d73cdf89-4461-4224-8cf1-53353a26afd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:43:26.074085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.074049 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv"] Nov 04 01:43:26.074690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.074666 4312 cpu_manager.go:414] "containerMap: removing container" podUID="30eb8587-68bf-46ee-90a9-d55b3aa61116" containerName="revoke-keys" Nov 04 01:43:26.074690 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.074687 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30eb8587-68bf-46ee-90a9-d55b3aa61116" containerName="revoke-keys" Nov 04 01:43:26.074904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.074830 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="30eb8587-68bf-46ee-90a9-d55b3aa61116" containerName="revoke-keys" Nov 04 01:43:26.077980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.077958 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.085239 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.085213 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:43:26.086224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.086201 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:43:26.088510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.088483 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv"] Nov 04 01:43:26.143217 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.143184 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:43:26.143217 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.143210 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:43:26.217988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.217948 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.217988 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.217985 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.218205 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.218082 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxcx\" (UniqueName: \"kubernetes.io/projected/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-kube-api-access-tsxcx\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.319012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.318971 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.319012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.319009 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.319238 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.319061 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxcx\" (UniqueName: \"kubernetes.io/projected/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-kube-api-access-tsxcx\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.319888 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.319864 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.321418 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.321397 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.331268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.331219 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxcx\" (UniqueName: \"kubernetes.io/projected/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-kube-api-access-tsxcx\") pod \"maas-api-revoke-sub-d2bc93bc-mr8fv\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.399277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.399243 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:26.705939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.705914 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv"] Nov 04 01:43:26.707985 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:43:26.707952 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d60fbb_8f8d_416b_bcb9_f9d9aa13ebe0.slice/crio-a0a3f9044fabdad44bfa4ee0f4887e449666e10cfbe55d7c990c9768d044f367 WatchSource:0}: Error finding container a0a3f9044fabdad44bfa4ee0f4887e449666e10cfbe55d7c990c9768d044f367: Status 404 returned error can't find the container with id a0a3f9044fabdad44bfa4ee0f4887e449666e10cfbe55d7c990c9768d044f367 Nov 04 01:43:26.960106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:26.960070 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" event={"ID":"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0","Type":"ContainerStarted","Data":"a0a3f9044fabdad44bfa4ee0f4887e449666e10cfbe55d7c990c9768d044f367"} Nov 04 01:43:27.967400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:27.967359 4312 generic.go:356] "Generic (PLEG): container finished" podID="86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" containerID="4b52ff98dcef80f6d402ef22f65b5f30c7a0298dbbf12d18c761cf151ef7dc22" exitCode=0 Nov 04 01:43:27.967791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:27.967450 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" event={"ID":"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0","Type":"ContainerDied","Data":"4b52ff98dcef80f6d402ef22f65b5f30c7a0298dbbf12d18c761cf151ef7dc22"} Nov 04 01:43:29.123259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.123232 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:29.249986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.249909 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-kube-api-access-tsxcx\" (UniqueName: \"kubernetes.io/projected/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-kube-api-access-tsxcx\") pod \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " Nov 04 01:43:29.249986 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.249949 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-openshift-ca\") pod \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " Nov 04 01:43:29.250234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.250048 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-xks-ca\") pod \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\" (UID: \"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0\") " Nov 04 01:43:29.250388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.250358 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-openshift-ca" pod "86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" (UID: "86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:29.252090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.252064 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-kube-api-access-tsxcx" pod "86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" (UID: "86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0"). InnerVolumeSpecName "kube-api-access-tsxcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:29.252090 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.252076 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-xks-ca" pod "86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" (UID: "86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:43:29.350979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.350951 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:29.350979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.350978 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxcx\" (UniqueName: \"kubernetes.io/projected/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-kube-api-access-tsxcx\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:29.351155 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.350990 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:29.456777 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:43:29.456742 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:43:29.979816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.979785 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" Nov 04 01:43:29.980021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.979827 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv" event={"ID":"86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0","Type":"ContainerDied","Data":"a0a3f9044fabdad44bfa4ee0f4887e449666e10cfbe55d7c990c9768d044f367"} Nov 04 01:43:29.980021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:29.979872 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a3f9044fabdad44bfa4ee0f4887e449666e10cfbe55d7c990c9768d044f367" Nov 04 01:43:38.231507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.231461 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b"] Nov 04 01:43:38.232015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.231912 4312 cpu_manager.go:414] "containerMap: removing container" podUID="86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" containerName="revoke-keys" Nov 04 01:43:38.232015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.231925 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" containerName="revoke-keys" Nov 04 01:43:38.232015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.231998 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" containerName="revoke-keys" Nov 04 01:43:38.235265 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.235248 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.248511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.248491 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-dbe634-openshift-default-dockercfg-glbz9\"" Nov 04 01:43:38.260221 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.260196 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b"] Nov 04 01:43:38.332912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.332867 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8703487-3195-44d9-9252-c07daf360c1a-istio-podinfo\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.332912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.332932 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8703487-3195-44d9-9252-c07daf360c1a-istiod-ca-cert\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.332912 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.332961 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-certs\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.333311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.333013 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-istio-token\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.333311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.333046 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-credential-socket\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.333311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.333088 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-socket\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.333311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.333128 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-data\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.333311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.333156 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62k9\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-kube-api-access-p62k9\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.333311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.333194 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-envoy\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434314 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8703487-3195-44d9-9252-c07daf360c1a-istiod-ca-cert\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434363 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434361 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-certs\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434395 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-istio-token\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434419 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-credential-socket\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434462 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-socket\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434503 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-data\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434530 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62k9\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-kube-api-access-p62k9\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.434628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434569 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-envoy\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.435002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434658 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8703487-3195-44d9-9252-c07daf360c1a-istio-podinfo\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.435002 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.434767 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-credential-socket\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.435111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.435017 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-certs\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.435168 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.435146 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-socket\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.435220 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.435159 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-data\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.435220 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.435181 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8703487-3195-44d9-9252-c07daf360c1a-istiod-ca-cert\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.437021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.436997 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-envoy\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.437208 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.437180 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8703487-3195-44d9-9252-c07daf360c1a-istio-podinfo\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.448593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.448567 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-istio-token\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.453180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.453153 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62k9\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-kube-api-access-p62k9\") pod \"e2e-worker-single-main-dbe634-openshift-default-69684858cck227b\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.548324 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.548298 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:38.841129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.841105 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b"] Nov 04 01:43:38.842423 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:43:38.842400 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8703487_3195_44d9_9252_c07daf360c1a.slice/crio-8411da1f205efbda92aaadad94fd22cd4bd03052abae878e7b5bb212ad550895 WatchSource:0}: Error finding container 8411da1f205efbda92aaadad94fd22cd4bd03052abae878e7b5bb212ad550895: Status 404 returned error can't find the container with id 8411da1f205efbda92aaadad94fd22cd4bd03052abae878e7b5bb212ad550895 Nov 04 01:43:38.845021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.844990 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:43:38.845103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.845060 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:43:38.845103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:38.845090 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:43:39.032171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:39.032135 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" event={"ID":"a8703487-3195-44d9-9252-c07daf360c1a","Type":"ContainerStarted","Data":"b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc"} Nov 04 01:43:39.032414 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:39.032195 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" event={"ID":"a8703487-3195-44d9-9252-c07daf360c1a","Type":"ContainerStarted","Data":"8411da1f205efbda92aaadad94fd22cd4bd03052abae878e7b5bb212ad550895"} Nov 04 01:43:39.066444 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:39.066395 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podStartSLOduration=1.066379557 podStartE2EDuration="1.066379557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:43:39.064065492 +0000 UTC m=+2451.086997517" watchObservedRunningTime="2026-11-04 01:43:39.066379557 +0000 UTC m=+2451.089311565" Nov 04 01:43:39.548624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:39.548584 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:39.553629 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:39.553604 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:40.037825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:40.037796 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:40.038672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:40.038651 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:42.661459 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.661427 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk"] Nov 04 01:43:42.665301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.665285 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.680298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.677745 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:43:42.680298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.678439 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:43:42.680298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.678528 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:43:42.680298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.678568 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rbkg2\"" Nov 04 01:43:42.680298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.678590 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-dbe634-serving-cert\"" Nov 04 01:43:42.680298 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.680223 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk"] Nov 04 01:43:42.697073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.697048 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm"] Nov 04 01:43:42.701438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.701414 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95"] Nov 04 01:43:42.704501 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.704465 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:42.705450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.705422 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:42.716720 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.716677 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm"] Nov 04 01:43:42.719138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.719115 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95"] Nov 04 01:43:42.734096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.734073 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ec4a373b-dockercfg-lzhcj\"" Nov 04 01:43:42.775550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.775515 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dbe634\"" Nov 04 01:43:42.779005 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.778977 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mphfv\" (UniqueName: \"kubernetes.io/projected/4e67a854-1a44-4507-8a74-44f872dec7c4-kube-api-access-mphfv\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:42.779127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.779015 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.779127 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.779049 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxx6\" (UniqueName: \"kubernetes.io/projected/27979e49-d376-4c23-bd0a-256fae584128-kube-api-access-8xxx6\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.779210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.779184 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c309629f-4e4e-4e39-8980-32b666683163-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:42.779210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.779206 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27979e49-d376-4c23-bd0a-256fae584128-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.779279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.779225 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e67a854-1a44-4507-8a74-44f872dec7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:42.779279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.779242 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.779395 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.779374 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rhxt\" (UniqueName: \"kubernetes.io/projected/c309629f-4e4e-4e39-8980-32b666683163-kube-api-access-5rhxt\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:42.880124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880084 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c309629f-4e4e-4e39-8980-32b666683163-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:42.880366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880148 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27979e49-d376-4c23-bd0a-256fae584128-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.880366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880193 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e67a854-1a44-4507-8a74-44f872dec7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:42.880366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880228 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.880366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880291 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhxt\" (UniqueName: \"kubernetes.io/projected/c309629f-4e4e-4e39-8980-32b666683163-kube-api-access-5rhxt\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:42.880366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880350 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mphfv\" (UniqueName: \"kubernetes.io/projected/4e67a854-1a44-4507-8a74-44f872dec7c4-kube-api-access-mphfv\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:42.880688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880379 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.880688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.880420 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxx6\" (UniqueName: \"kubernetes.io/projected/27979e49-d376-4c23-bd0a-256fae584128-kube-api-access-8xxx6\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.883569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.881117 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27979e49-d376-4c23-bd0a-256fae584128-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.883569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.881560 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c309629f-4e4e-4e39-8980-32b666683163-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:42.884657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.884460 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.887535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.887506 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e67a854-1a44-4507-8a74-44f872dec7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:42.889254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.889231 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.894542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.894523 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhxt\" (UniqueName: \"kubernetes.io/projected/c309629f-4e4e-4e39-8980-32b666683163-kube-api-access-5rhxt\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:42.894710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.894688 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxx6\" (UniqueName: \"kubernetes.io/projected/27979e49-d376-4c23-bd0a-256fae584128-kube-api-access-8xxx6\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:42.911226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.911208 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mphfv\" (UniqueName: \"kubernetes.io/projected/4e67a854-1a44-4507-8a74-44f872dec7c4-kube-api-access-mphfv\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:42.981091 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:42.981025 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:43.025725 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:43.025691 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:43.035163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:43.035114 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:43.281566 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:43.281532 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk"] Nov 04 01:43:43.283435 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:43:43.283404 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27979e49_d376_4c23_bd0a_256fae584128.slice/crio-3e5d467e6e15e23ad728fc0d33a7ac95bf5bf4c7a09d4686e13aa3d2292b935f WatchSource:0}: Error finding container 3e5d467e6e15e23ad728fc0d33a7ac95bf5bf4c7a09d4686e13aa3d2292b935f: Status 404 returned error can't find the container with id 3e5d467e6e15e23ad728fc0d33a7ac95bf5bf4c7a09d4686e13aa3d2292b935f Nov 04 01:43:43.340132 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:43.340101 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm"] Nov 04 01:43:43.342061 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:43:43.342028 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc309629f_4e4e_4e39_8980_32b666683163.slice/crio-da67a13f85a42763391ec83bb182429e4f26fe39015780c7a465ad792c88281a WatchSource:0}: Error finding container da67a13f85a42763391ec83bb182429e4f26fe39015780c7a465ad792c88281a: Status 404 returned error can't find the container with id da67a13f85a42763391ec83bb182429e4f26fe39015780c7a465ad792c88281a Nov 04 01:43:43.347904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:43.347878 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95"] Nov 04 01:43:43.348796 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:43:43.348768 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e67a854_1a44_4507_8a74_44f872dec7c4.slice/crio-9030a35b386d5577be43129e5fdfac71999da9cf5bd9b4d775aea1eb7d1e9168 WatchSource:0}: Error finding container 9030a35b386d5577be43129e5fdfac71999da9cf5bd9b4d775aea1eb7d1e9168: Status 404 returned error can't find the container with id 9030a35b386d5577be43129e5fdfac71999da9cf5bd9b4d775aea1eb7d1e9168 Nov 04 01:43:44.051439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.051406 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5"] Nov 04 01:43:44.060290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.060263 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p62f5"] Nov 04 01:43:44.071593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.071561 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" event={"ID":"4e67a854-1a44-4507-8a74-44f872dec7c4","Type":"ContainerStarted","Data":"98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c"} Nov 04 01:43:44.071745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.071640 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:44.071745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.071667 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" event={"ID":"4e67a854-1a44-4507-8a74-44f872dec7c4","Type":"ContainerStarted","Data":"9030a35b386d5577be43129e5fdfac71999da9cf5bd9b4d775aea1eb7d1e9168"} Nov 04 01:43:44.073350 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.073310 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" event={"ID":"c309629f-4e4e-4e39-8980-32b666683163","Type":"ContainerStarted","Data":"9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e"} Nov 04 01:43:44.073449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.073394 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:44.073449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.073420 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" event={"ID":"c309629f-4e4e-4e39-8980-32b666683163","Type":"ContainerStarted","Data":"da67a13f85a42763391ec83bb182429e4f26fe39015780c7a465ad792c88281a"} Nov 04 01:43:44.075018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.074991 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" event={"ID":"27979e49-d376-4c23-bd0a-256fae584128","Type":"ContainerStarted","Data":"3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819"} Nov 04 01:43:44.075109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.075032 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" event={"ID":"27979e49-d376-4c23-bd0a-256fae584128","Type":"ContainerStarted","Data":"3e5d467e6e15e23ad728fc0d33a7ac95bf5bf4c7a09d4686e13aa3d2292b935f"} Nov 04 01:43:44.075109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.075085 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:44.108781 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.108754 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fadec4-9d24-457c-950f-df2c156cb7bc" path="/var/lib/kubelet/pods/42fadec4-9d24-457c-950f-df2c156cb7bc/volumes" Nov 04 01:43:44.121041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.120999 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" podStartSLOduration=2.120987243 podStartE2EDuration="2.120987243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:43:44.119810378 +0000 UTC m=+2456.142742413" watchObservedRunningTime="2026-11-04 01:43:44.120987243 +0000 UTC m=+2456.143919250" Nov 04 01:43:44.160450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:44.160384 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" podStartSLOduration=2.160370453 podStartE2EDuration="2.160370453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:43:44.158287858 +0000 UTC m=+2456.181219864" watchObservedRunningTime="2026-11-04 01:43:44.160370453 +0000 UTC m=+2456.183302459" Nov 04 01:43:50.088244 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:50.088212 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:43:50.114688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:50.114638 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" podStartSLOduration=7.630415839 podStartE2EDuration="8.114625429s" totalImagesPullingTime="484.20959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:43:44.186478655 +0000 UTC m=+2456.209410662" watchObservedRunningTime="2026-11-04 01:43:50.114625429 +0000 UTC m=+2462.137557435" Nov 04 01:43:50.773400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:50.773365 4312 scope.go:122] "RemoveContainer" containerID="285674212df471aecfb6464fddca9effe87d552573e2cf3b138b143865ac9d3c" Nov 04 01:43:50.782438 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:50.782316 4312 scope.go:122] "RemoveContainer" containerID="23b3e77c165609b62d0bafad210d9f3262fb62d264e79f73657972abd80d85e1" Nov 04 01:43:50.791111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:50.791091 4312 scope.go:122] "RemoveContainer" containerID="2f4d2caceb1f89ccf774dfbc29f434f88a393b139fef08f5a07adc4236aba9a9" Nov 04 01:43:54.048111 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:54.048054 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp"] Nov 04 01:43:54.050290 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:54.050261 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k2jdp"] Nov 04 01:43:54.107271 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:54.107240 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2562462f-1916-4aa4-960b-51c804005ee7" path="/var/lib/kubelet/pods/2562462f-1916-4aa4-960b-51c804005ee7/volumes" Nov 04 01:43:55.040032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.039996 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 04 01:43:55.040251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.040062 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 04 01:43:55.086453 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.086425 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:43:55.086939 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.086629 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:43:55.143599 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.143566 4312 generic.go:356] "Generic (PLEG): container finished" podID="a8703487-3195-44d9-9252-c07daf360c1a" containerID="b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc" exitCode=0 Nov 04 01:43:55.143760 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.143648 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" event={"ID":"a8703487-3195-44d9-9252-c07daf360c1a","Type":"ContainerDied","Data":"b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc"} Nov 04 01:43:55.144207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.144185 4312 scope.go:122] "RemoveContainer" containerID="b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc" Nov 04 01:43:55.145847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.145809 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:43:55.145954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.145907 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:43:55.146549 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:55.146499 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:43:56.176847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:56.176808 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" event={"ID":"a8703487-3195-44d9-9252-c07daf360c1a","Type":"ContainerStarted","Data":"aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e"} Nov 04 01:43:56.560356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:56.554714 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:43:56.560356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:56.554846 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 04 01:43:56.560356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:56.554892 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 04 01:43:57.559835 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:57.559798 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 04 01:43:57.560462 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:57.560430 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 04 01:43:58.120328 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.120296 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7"] Nov 04 01:43:58.124106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.124089 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.129876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.129850 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7"] Nov 04 01:43:58.134150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.133809 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dbe634\"/\"kube-root-ca.crt\"" Nov 04 01:43:58.134150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.133864 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dbe634\"/\"openshift-service-ca.crt\"" Nov 04 01:43:58.134150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.134008 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dbe634\"/\"default-dockercfg-7fmrs\"" Nov 04 01:43:58.134150 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.134082 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dbe634\"/\"simulator-single-main-dbe634-kserve-self-signed-certs\"" Nov 04 01:43:58.291543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.291511 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-tmp-dir\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.291734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.291574 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-dshm\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.291782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.291756 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c528bc0-973b-4641-b75d-16abbde9d192-tls-certs\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.291867 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.291797 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-model-cache\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.291913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.291890 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-home\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.292013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.291994 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtdz\" (UniqueName: \"kubernetes.io/projected/3c528bc0-973b-4641-b75d-16abbde9d192-kube-api-access-vxtdz\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.392978 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtdz\" (UniqueName: \"kubernetes.io/projected/3c528bc0-973b-4641-b75d-16abbde9d192-kube-api-access-vxtdz\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.393038 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-tmp-dir\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.393094 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-dshm\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.393171 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c528bc0-973b-4641-b75d-16abbde9d192-tls-certs\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.393206 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-model-cache\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.393270 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-home\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.393728 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-home\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.395639 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.395243 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-tmp-dir\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.397713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.397683 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-model-cache\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.403185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.402510 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-dshm\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.403185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.403072 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c528bc0-973b-4641-b75d-16abbde9d192-tls-certs\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.412047 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.412005 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtdz\" (UniqueName: \"kubernetes.io/projected/3c528bc0-973b-4641-b75d-16abbde9d192-kube-api-access-vxtdz\") pod \"simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.439929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.439903 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:58.572357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.569770 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 04 01:43:58.572357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.569834 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 04 01:43:58.769027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:58.765695 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7"] Nov 04 01:43:59.195008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:59.194972 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" event={"ID":"3c528bc0-973b-4641-b75d-16abbde9d192","Type":"ContainerStarted","Data":"2066d9a4d7bf73699190384818a3ce727ea697bf4450eac8a2ba289560f1605b"} Nov 04 01:43:59.195175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:59.195020 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" event={"ID":"3c528bc0-973b-4641-b75d-16abbde9d192","Type":"ContainerStarted","Data":"68bb3c9a7fec078d4ec2bf4bd9e7dbe7efe4f39993cbb99de1dcc955ad219d94"} Nov 04 01:43:59.266618 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:59.266550 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" podStartSLOduration=1.266528205 podStartE2EDuration="1.266528205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:43:59.255796232 +0000 UTC m=+2471.278728263" watchObservedRunningTime="2026-11-04 01:43:59.266528205 +0000 UTC m=+2471.289460214" Nov 04 01:43:59.440698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:59.440665 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:43:59.490795 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:43:59.490695 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:43:59.558012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:59.557437 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 04 01:43:59.558012 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:43:59.557497 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 04 01:44:00.552358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:00.548749 4312 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 04 01:44:00.552358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:00.548815 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 04 01:44:01.727700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:01.727666 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:44:01.728425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:01.728388 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:44:01.729593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:01.729571 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:44:08.441010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:08.440974 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:44:08.453755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:08.453729 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:44:09.258885 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:09.258844 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:44:29.460073 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:44:29.460032 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:44:44.389116 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:44.389081 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7"] Nov 04 01:44:45.427291 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:45.427256 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm"] Nov 04 01:44:45.439379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:45.439343 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95"] Nov 04 01:44:46.580507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.580472 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk"] Nov 04 01:44:46.581479 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.581445 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" podUID="27979e49-d376-4c23-bd0a-256fae584128" containerName="maas-api" containerID="cri-o://3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819" gracePeriod=30 Nov 04 01:44:46.859096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.859072 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:44:46.901393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.901358 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-metrics-certs\") pod \"27979e49-d376-4c23-bd0a-256fae584128\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " Nov 04 01:44:46.901586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.901429 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-maas-api-tls\") pod \"27979e49-d376-4c23-bd0a-256fae584128\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " Nov 04 01:44:46.901586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.901468 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27979e49-d376-4c23-bd0a-256fae584128-kube-api-access-8xxx6\" (UniqueName: \"kubernetes.io/projected/27979e49-d376-4c23-bd0a-256fae584128-kube-api-access-8xxx6\") pod \"27979e49-d376-4c23-bd0a-256fae584128\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " Nov 04 01:44:46.901586 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.901543 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27979e49-d376-4c23-bd0a-256fae584128-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27979e49-d376-4c23-bd0a-256fae584128-openshift-service-ca\") pod \"27979e49-d376-4c23-bd0a-256fae584128\" (UID: \"27979e49-d376-4c23-bd0a-256fae584128\") " Nov 04 01:44:46.902207 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.902174 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27979e49-d376-4c23-bd0a-256fae584128-openshift-service-ca" pod "27979e49-d376-4c23-bd0a-256fae584128" (UID: "27979e49-d376-4c23-bd0a-256fae584128"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:46.903829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.903791 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-metrics-certs" pod "27979e49-d376-4c23-bd0a-256fae584128" (UID: "27979e49-d376-4c23-bd0a-256fae584128"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:46.904093 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.904068 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27979e49-d376-4c23-bd0a-256fae584128-kube-api-access-8xxx6" pod "27979e49-d376-4c23-bd0a-256fae584128" (UID: "27979e49-d376-4c23-bd0a-256fae584128"). InnerVolumeSpecName "kube-api-access-8xxx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:46.904188 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:46.904142 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-maas-api-tls" pod "27979e49-d376-4c23-bd0a-256fae584128" (UID: "27979e49-d376-4c23-bd0a-256fae584128"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:47.003568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.003535 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:47.003568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.003571 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27979e49-d376-4c23-bd0a-256fae584128-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:47.003809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.003621 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xxx6\" (UniqueName: \"kubernetes.io/projected/27979e49-d376-4c23-bd0a-256fae584128-kube-api-access-8xxx6\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:47.003809 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.003637 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27979e49-d376-4c23-bd0a-256fae584128-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:47.469449 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.469413 4312 generic.go:356] "Generic (PLEG): container finished" podID="27979e49-d376-4c23-bd0a-256fae584128" containerID="3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819" exitCode=0 Nov 04 01:44:47.469644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.469460 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" Nov 04 01:44:47.469644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.469498 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" event={"ID":"27979e49-d376-4c23-bd0a-256fae584128","Type":"ContainerDied","Data":"3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819"} Nov 04 01:44:47.469644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.469556 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk" event={"ID":"27979e49-d376-4c23-bd0a-256fae584128","Type":"ContainerDied","Data":"3e5d467e6e15e23ad728fc0d33a7ac95bf5bf4c7a09d4686e13aa3d2292b935f"} Nov 04 01:44:47.469644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.469587 4312 scope.go:122] "RemoveContainer" containerID="3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819" Nov 04 01:44:47.479749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.479729 4312 scope.go:122] "RemoveContainer" containerID="3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819" Nov 04 01:44:47.480039 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:44:47.480019 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819\": container with ID starting with 3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819 not found: ID does not exist" containerID="3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819" Nov 04 01:44:47.480091 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.480052 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819"} err="failed to get container status \"3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819\": rpc error: code = NotFound desc = could not find container \"3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819\": container with ID starting with 3adbe06bb4290ed4e02bda5a4358e1f322e82cf7b7897c8d0eabd93c866cc819 not found: ID does not exist" Nov 04 01:44:47.492535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.492509 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk"] Nov 04 01:44:47.497001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:47.496968 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-mxmdk"] Nov 04 01:44:48.108926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:48.108890 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27979e49-d376-4c23-bd0a-256fae584128" path="/var/lib/kubelet/pods/27979e49-d376-4c23-bd0a-256fae584128/volumes" Nov 04 01:44:50.427714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.427668 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" podUID="c309629f-4e4e-4e39-8980-32b666683163" containerName="payload-processing" containerID="cri-o://9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e" gracePeriod=25 Nov 04 01:44:50.440357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.440290 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" podUID="4e67a854-1a44-4507-8a74-44f872dec7c4" containerName="payload-pre-processing" containerID="cri-o://98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c" gracePeriod=25 Nov 04 01:44:50.714454 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.714429 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:44:50.746349 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.746315 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:44:50.818935 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.818903 4312 scope.go:122] "RemoveContainer" containerID="73d1425776334914642ce15cb46699be4d2d5307e06fbb273442d737dd606a16" Nov 04 01:44:50.828182 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.828162 4312 scope.go:122] "RemoveContainer" containerID="d2a7d98f458652c248e5f9b1dc8cf57afa5a900bee77d6ad9dca2d5afd2f54f0" Nov 04 01:44:50.836225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.836204 4312 scope.go:122] "RemoveContainer" containerID="1e3e9a7d5774f0bf77125e335c0cba8309d7f0d762644c650f1267079944ae95" Nov 04 01:44:50.841466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.841443 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e67a854-1a44-4507-8a74-44f872dec7c4-kube-api-access-mphfv\" (UniqueName: \"kubernetes.io/projected/4e67a854-1a44-4507-8a74-44f872dec7c4-kube-api-access-mphfv\") pod \"4e67a854-1a44-4507-8a74-44f872dec7c4\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " Nov 04 01:44:50.841564 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.841522 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c309629f-4e4e-4e39-8980-32b666683163-kube-api-access-5rhxt\" (UniqueName: \"kubernetes.io/projected/c309629f-4e4e-4e39-8980-32b666683163-kube-api-access-5rhxt\") pod \"c309629f-4e4e-4e39-8980-32b666683163\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " Nov 04 01:44:50.841680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.841664 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c309629f-4e4e-4e39-8980-32b666683163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c309629f-4e4e-4e39-8980-32b666683163-plugins-config-volume\") pod \"c309629f-4e4e-4e39-8980-32b666683163\" (UID: \"c309629f-4e4e-4e39-8980-32b666683163\") " Nov 04 01:44:50.841733 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.841724 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e67a854-1a44-4507-8a74-44f872dec7c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e67a854-1a44-4507-8a74-44f872dec7c4-plugins-config-volume\") pod \"4e67a854-1a44-4507-8a74-44f872dec7c4\" (UID: \"4e67a854-1a44-4507-8a74-44f872dec7c4\") " Nov 04 01:44:50.842178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.842092 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c309629f-4e4e-4e39-8980-32b666683163-plugins-config-volume" pod "c309629f-4e4e-4e39-8980-32b666683163" (UID: "c309629f-4e4e-4e39-8980-32b666683163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:50.842178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.842120 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e67a854-1a44-4507-8a74-44f872dec7c4-plugins-config-volume" pod "4e67a854-1a44-4507-8a74-44f872dec7c4" (UID: "4e67a854-1a44-4507-8a74-44f872dec7c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:50.843633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.843581 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c309629f-4e4e-4e39-8980-32b666683163-kube-api-access-5rhxt" pod "c309629f-4e4e-4e39-8980-32b666683163" (UID: "c309629f-4e4e-4e39-8980-32b666683163"). InnerVolumeSpecName "kube-api-access-5rhxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:50.843854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.843832 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e67a854-1a44-4507-8a74-44f872dec7c4-kube-api-access-mphfv" pod "4e67a854-1a44-4507-8a74-44f872dec7c4" (UID: "4e67a854-1a44-4507-8a74-44f872dec7c4"). InnerVolumeSpecName "kube-api-access-mphfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:50.844872 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.844852 4312 scope.go:122] "RemoveContainer" containerID="052a660ba2bd9b6f0ad0cc489adaefafc94958d005c18b072d1a2edc2119c2ea" Nov 04 01:44:50.852374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.852357 4312 scope.go:122] "RemoveContainer" containerID="fb4b08e52fedb6885050e35aeb7fedef9d2fd37333ba8180a155e74d95a3e59a" Nov 04 01:44:50.942672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.942612 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c309629f-4e4e-4e39-8980-32b666683163-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:50.942672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.942636 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e67a854-1a44-4507-8a74-44f872dec7c4-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:50.942672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.942646 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mphfv\" (UniqueName: \"kubernetes.io/projected/4e67a854-1a44-4507-8a74-44f872dec7c4-kube-api-access-mphfv\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:50.942672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:50.942656 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rhxt\" (UniqueName: \"kubernetes.io/projected/c309629f-4e4e-4e39-8980-32b666683163-kube-api-access-5rhxt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:51.495905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.495813 4312 generic.go:356] "Generic (PLEG): container finished" podID="c309629f-4e4e-4e39-8980-32b666683163" containerID="9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e" exitCode=0 Nov 04 01:44:51.495905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.495872 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" Nov 04 01:44:51.496465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.495905 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" event={"ID":"c309629f-4e4e-4e39-8980-32b666683163","Type":"ContainerDied","Data":"9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e"} Nov 04 01:44:51.496465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.495959 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm" event={"ID":"c309629f-4e4e-4e39-8980-32b666683163","Type":"ContainerDied","Data":"da67a13f85a42763391ec83bb182429e4f26fe39015780c7a465ad792c88281a"} Nov 04 01:44:51.496465 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.496001 4312 scope.go:122] "RemoveContainer" containerID="9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e" Nov 04 01:44:51.497729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.497704 4312 generic.go:356] "Generic (PLEG): container finished" podID="4e67a854-1a44-4507-8a74-44f872dec7c4" containerID="98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c" exitCode=0 Nov 04 01:44:51.497883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.497738 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" Nov 04 01:44:51.497883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.497769 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" event={"ID":"4e67a854-1a44-4507-8a74-44f872dec7c4","Type":"ContainerDied","Data":"98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c"} Nov 04 01:44:51.497883 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.497834 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95" event={"ID":"4e67a854-1a44-4507-8a74-44f872dec7c4","Type":"ContainerDied","Data":"9030a35b386d5577be43129e5fdfac71999da9cf5bd9b4d775aea1eb7d1e9168"} Nov 04 01:44:51.506219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.506202 4312 scope.go:122] "RemoveContainer" containerID="9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e" Nov 04 01:44:51.506521 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:44:51.506499 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e\": container with ID starting with 9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e not found: ID does not exist" containerID="9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e" Nov 04 01:44:51.506608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.506532 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e"} err="failed to get container status \"9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e\": rpc error: code = NotFound desc = could not find container \"9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e\": container with ID starting with 9b9256405e6a03bcf2c5bf4dde9fe8837ddfc47633e100cf135192069b943e8e not found: ID does not exist" Nov 04 01:44:51.506608 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.506554 4312 scope.go:122] "RemoveContainer" containerID="98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c" Nov 04 01:44:51.514285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.514269 4312 scope.go:122] "RemoveContainer" containerID="98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c" Nov 04 01:44:51.514551 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:44:51.514535 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c\": container with ID starting with 98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c not found: ID does not exist" containerID="98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c" Nov 04 01:44:51.514603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.514556 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c"} err="failed to get container status \"98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c\": rpc error: code = NotFound desc = could not find container \"98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c\": container with ID starting with 98614ab13fd08f04a078a96c905f2621fc45ed39ac0783d315591e8a2683377c not found: ID does not exist" Nov 04 01:44:51.520059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.520036 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm"] Nov 04 01:44:51.523775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.523752 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dwnjcm"] Nov 04 01:44:51.537135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.537111 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95"] Nov 04 01:44:51.541003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:51.540983 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b62jn95"] Nov 04 01:44:52.107598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:52.107568 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e67a854-1a44-4507-8a74-44f872dec7c4" path="/var/lib/kubelet/pods/4e67a854-1a44-4507-8a74-44f872dec7c4/volumes" Nov 04 01:44:52.107977 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:52.107964 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c309629f-4e4e-4e39-8980-32b666683163" path="/var/lib/kubelet/pods/c309629f-4e4e-4e39-8980-32b666683163/volumes" Nov 04 01:44:59.409019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.408976 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" podUID="3c528bc0-973b-4641-b75d-16abbde9d192" containerName="main" containerID="cri-o://2066d9a4d7bf73699190384818a3ce727ea697bf4450eac8a2ba289560f1605b" gracePeriod=45 Nov 04 01:44:59.456515 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:44:59.456472 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:44:59.540007 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.539980 4312 generic.go:356] "Generic (PLEG): container finished" podID="3c528bc0-973b-4641-b75d-16abbde9d192" containerID="2066d9a4d7bf73699190384818a3ce727ea697bf4450eac8a2ba289560f1605b" exitCode=0 Nov 04 01:44:59.540142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.540050 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" event={"ID":"3c528bc0-973b-4641-b75d-16abbde9d192","Type":"ContainerDied","Data":"2066d9a4d7bf73699190384818a3ce727ea697bf4450eac8a2ba289560f1605b"} Nov 04 01:44:59.659540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.659480 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:44:59.730285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730255 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-model-cache\") pod \"3c528bc0-973b-4641-b75d-16abbde9d192\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " Nov 04 01:44:59.730472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730322 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c528bc0-973b-4641-b75d-16abbde9d192-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c528bc0-973b-4641-b75d-16abbde9d192-tls-certs\") pod \"3c528bc0-973b-4641-b75d-16abbde9d192\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " Nov 04 01:44:59.730472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730370 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-tmp-dir\") pod \"3c528bc0-973b-4641-b75d-16abbde9d192\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " Nov 04 01:44:59.730472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730447 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c528bc0-973b-4641-b75d-16abbde9d192-kube-api-access-vxtdz\" (UniqueName: \"kubernetes.io/projected/3c528bc0-973b-4641-b75d-16abbde9d192-kube-api-access-vxtdz\") pod \"3c528bc0-973b-4641-b75d-16abbde9d192\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " Nov 04 01:44:59.730472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730466 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-home\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-home\") pod \"3c528bc0-973b-4641-b75d-16abbde9d192\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " Nov 04 01:44:59.730713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730546 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-dshm\") pod \"3c528bc0-973b-4641-b75d-16abbde9d192\" (UID: \"3c528bc0-973b-4641-b75d-16abbde9d192\") " Nov 04 01:44:59.730713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730600 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-model-cache" pod "3c528bc0-973b-4641-b75d-16abbde9d192" (UID: "3c528bc0-973b-4641-b75d-16abbde9d192"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:59.730817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730749 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-tmp-dir" pod "3c528bc0-973b-4641-b75d-16abbde9d192" (UID: "3c528bc0-973b-4641-b75d-16abbde9d192"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:59.730817 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730788 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-home" pod "3c528bc0-973b-4641-b75d-16abbde9d192" (UID: "3c528bc0-973b-4641-b75d-16abbde9d192"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:59.730968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730950 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:59.731030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730975 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:59.731030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.730989 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:59.732678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.732653 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c528bc0-973b-4641-b75d-16abbde9d192-kube-api-access-vxtdz" pod "3c528bc0-973b-4641-b75d-16abbde9d192" (UID: "3c528bc0-973b-4641-b75d-16abbde9d192"). InnerVolumeSpecName "kube-api-access-vxtdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:59.732970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.732945 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-dshm" pod "3c528bc0-973b-4641-b75d-16abbde9d192" (UID: "3c528bc0-973b-4641-b75d-16abbde9d192"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:59.733052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.732989 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c528bc0-973b-4641-b75d-16abbde9d192-tls-certs" pod "3c528bc0-973b-4641-b75d-16abbde9d192" (UID: "3c528bc0-973b-4641-b75d-16abbde9d192"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:59.832603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.832571 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c528bc0-973b-4641-b75d-16abbde9d192-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:59.832603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.832600 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtdz\" (UniqueName: \"kubernetes.io/projected/3c528bc0-973b-4641-b75d-16abbde9d192-kube-api-access-vxtdz\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:59.832603 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:44:59.832611 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c528bc0-973b-4641-b75d-16abbde9d192-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:00.136598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.136569 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt"] Nov 04 01:45:00.136987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.136975 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4e67a854-1a44-4507-8a74-44f872dec7c4" containerName="payload-pre-processing" Nov 04 01:45:00.137027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.136988 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e67a854-1a44-4507-8a74-44f872dec7c4" containerName="payload-pre-processing" Nov 04 01:45:00.137027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137005 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3c528bc0-973b-4641-b75d-16abbde9d192" containerName="main" Nov 04 01:45:00.137027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137010 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c528bc0-973b-4641-b75d-16abbde9d192" containerName="main" Nov 04 01:45:00.137126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137030 4312 cpu_manager.go:414] "containerMap: removing container" podUID="c309629f-4e4e-4e39-8980-32b666683163" containerName="payload-processing" Nov 04 01:45:00.137126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137035 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c309629f-4e4e-4e39-8980-32b666683163" containerName="payload-processing" Nov 04 01:45:00.137126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137048 4312 cpu_manager.go:414] "containerMap: removing container" podUID="27979e49-d376-4c23-bd0a-256fae584128" containerName="maas-api" Nov 04 01:45:00.137126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137053 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27979e49-d376-4c23-bd0a-256fae584128" containerName="maas-api" Nov 04 01:45:00.137126 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137119 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="c309629f-4e4e-4e39-8980-32b666683163" containerName="payload-processing" Nov 04 01:45:00.137280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137128 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="27979e49-d376-4c23-bd0a-256fae584128" containerName="maas-api" Nov 04 01:45:00.137280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137136 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e67a854-1a44-4507-8a74-44f872dec7c4" containerName="payload-pre-processing" Nov 04 01:45:00.137280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.137143 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c528bc0-973b-4641-b75d-16abbde9d192" containerName="main" Nov 04 01:45:00.140468 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.140449 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.147628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.147241 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:45:00.147987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.147908 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:45:00.147987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.147940 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:45:00.149993 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.149969 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt"] Nov 04 01:45:00.236834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.236802 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gq65\" (UniqueName: \"kubernetes.io/projected/dca598ee-3551-4f28-b64c-8335660d13ce-kube-api-access-5gq65\") pod \"maas-api-key-cleanup-29895945-8h7dt\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.237042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.236878 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca598ee-3551-4f28-b64c-8335660d13ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-8h7dt\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.337956 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.337925 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gq65\" (UniqueName: \"kubernetes.io/projected/dca598ee-3551-4f28-b64c-8335660d13ce-kube-api-access-5gq65\") pod \"maas-api-key-cleanup-29895945-8h7dt\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.338183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.337992 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca598ee-3551-4f28-b64c-8335660d13ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-8h7dt\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.338645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.338626 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca598ee-3551-4f28-b64c-8335660d13ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-8h7dt\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.350648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.350625 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gq65\" (UniqueName: \"kubernetes.io/projected/dca598ee-3551-4f28-b64c-8335660d13ce-kube-api-access-5gq65\") pod \"maas-api-key-cleanup-29895945-8h7dt\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.461296 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.461237 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:00.546463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.546433 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" Nov 04 01:45:00.546626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.546432 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7" event={"ID":"3c528bc0-973b-4641-b75d-16abbde9d192","Type":"ContainerDied","Data":"68bb3c9a7fec078d4ec2bf4bd9e7dbe7efe4f39993cbb99de1dcc955ad219d94"} Nov 04 01:45:00.546626 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.546566 4312 scope.go:122] "RemoveContainer" containerID="2066d9a4d7bf73699190384818a3ce727ea697bf4450eac8a2ba289560f1605b" Nov 04 01:45:00.570870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.570839 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7"] Nov 04 01:45:00.575957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.575936 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe634/simulator-single-main-dbe634-kserve-6d85f9b644-8qqw7"] Nov 04 01:45:00.735640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:00.735608 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt"] Nov 04 01:45:00.737790 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:45:00.737767 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca598ee_3551_4f28_b64c_8335660d13ce.slice/crio-3a2324ed68ca4f72bc7d517fefa81e65421f6e782672dd8db78ab3879afae377 WatchSource:0}: Error finding container 3a2324ed68ca4f72bc7d517fefa81e65421f6e782672dd8db78ab3879afae377: Status 404 returned error can't find the container with id 3a2324ed68ca4f72bc7d517fefa81e65421f6e782672dd8db78ab3879afae377 Nov 04 01:45:01.554033 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:01.554002 4312 generic.go:356] "Generic (PLEG): container finished" podID="dca598ee-3551-4f28-b64c-8335660d13ce" containerID="ebc89199e7a3b8e93ec20d3229712c77859835e45dfa0b6a57e4cb6403664759" exitCode=60 Nov 04 01:45:01.554455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:01.554068 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" event={"ID":"dca598ee-3551-4f28-b64c-8335660d13ce","Type":"ContainerDied","Data":"ebc89199e7a3b8e93ec20d3229712c77859835e45dfa0b6a57e4cb6403664759"} Nov 04 01:45:01.554455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:01.554108 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" event={"ID":"dca598ee-3551-4f28-b64c-8335660d13ce","Type":"ContainerStarted","Data":"3a2324ed68ca4f72bc7d517fefa81e65421f6e782672dd8db78ab3879afae377"} Nov 04 01:45:01.554455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:01.554315 4312 scope.go:122] "RemoveContainer" containerID="ebc89199e7a3b8e93ec20d3229712c77859835e45dfa0b6a57e4cb6403664759" Nov 04 01:45:02.107741 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:02.107710 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c528bc0-973b-4641-b75d-16abbde9d192" path="/var/lib/kubelet/pods/3c528bc0-973b-4641-b75d-16abbde9d192/volumes" Nov 04 01:45:02.377813 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:02.377749 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca598ee_3551_4f28_b64c_8335660d13ce.slice/crio-5472ec81de43de6479dfbd5df6474962028d993afe28f145711ea8104b892dd1.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:45:02.561851 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:02.561820 4312 generic.go:356] "Generic (PLEG): container finished" podID="dca598ee-3551-4f28-b64c-8335660d13ce" containerID="5472ec81de43de6479dfbd5df6474962028d993afe28f145711ea8104b892dd1" exitCode=60 Nov 04 01:45:02.562246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:02.561903 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" event={"ID":"dca598ee-3551-4f28-b64c-8335660d13ce","Type":"ContainerDied","Data":"5472ec81de43de6479dfbd5df6474962028d993afe28f145711ea8104b892dd1"} Nov 04 01:45:02.562246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:02.561960 4312 scope.go:122] "RemoveContainer" containerID="ebc89199e7a3b8e93ec20d3229712c77859835e45dfa0b6a57e4cb6403664759" Nov 04 01:45:02.562246 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:02.562180 4312 scope.go:122] "RemoveContainer" containerID="5472ec81de43de6479dfbd5df6474962028d993afe28f145711ea8104b892dd1" Nov 04 01:45:02.562480 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:02.562462 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-8h7dt_odh-ai-gateway-infra(dca598ee-3551-4f28-b64c-8335660d13ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" Nov 04 01:45:03.569393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:03.569365 4312 scope.go:122] "RemoveContainer" containerID="5472ec81de43de6479dfbd5df6474962028d993afe28f145711ea8104b892dd1" Nov 04 01:45:03.569812 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:03.569562 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-8h7dt_odh-ai-gateway-infra(dca598ee-3551-4f28-b64c-8335660d13ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" Nov 04 01:45:14.100880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:14.100853 4312 scope.go:122] "RemoveContainer" containerID="5472ec81de43de6479dfbd5df6474962028d993afe28f145711ea8104b892dd1" Nov 04 01:45:14.629377 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:14.629276 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" event={"ID":"dca598ee-3551-4f28-b64c-8335660d13ce","Type":"ContainerStarted","Data":"3e99bc93b9e360aa4b07f9116358443a5f0248450ba1fc80c2727ed7facfaba7"} Nov 04 01:45:14.649598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:14.649550 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" podStartSLOduration=14.649536615 podStartE2EDuration="14.649536615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:14.647645377 +0000 UTC m=+2546.670577397" watchObservedRunningTime="2026-11-04 01:45:14.649536615 +0000 UTC m=+2546.672468671" Nov 04 01:45:15.636098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:15.636066 4312 generic.go:356] "Generic (PLEG): container finished" podID="dca598ee-3551-4f28-b64c-8335660d13ce" containerID="3e99bc93b9e360aa4b07f9116358443a5f0248450ba1fc80c2727ed7facfaba7" exitCode=60 Nov 04 01:45:15.636585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:15.636139 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" event={"ID":"dca598ee-3551-4f28-b64c-8335660d13ce","Type":"ContainerDied","Data":"3e99bc93b9e360aa4b07f9116358443a5f0248450ba1fc80c2727ed7facfaba7"} Nov 04 01:45:15.636585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:15.636201 4312 scope.go:122] "RemoveContainer" containerID="5472ec81de43de6479dfbd5df6474962028d993afe28f145711ea8104b892dd1" Nov 04 01:45:15.636585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:15.636551 4312 scope.go:122] "RemoveContainer" containerID="3e99bc93b9e360aa4b07f9116358443a5f0248450ba1fc80c2727ed7facfaba7" Nov 04 01:45:15.636868 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:15.636847 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-8h7dt_odh-ai-gateway-infra(dca598ee-3551-4f28-b64c-8335660d13ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" Nov 04 01:45:15.667248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:15.667222 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt"] Nov 04 01:45:16.783020 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:16.782994 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:16.883878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:16.883839 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca598ee-3551-4f28-b64c-8335660d13ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca598ee-3551-4f28-b64c-8335660d13ce-maas-api-service-ca\") pod \"dca598ee-3551-4f28-b64c-8335660d13ce\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " Nov 04 01:45:16.884084 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:16.883907 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca598ee-3551-4f28-b64c-8335660d13ce-kube-api-access-5gq65\" (UniqueName: \"kubernetes.io/projected/dca598ee-3551-4f28-b64c-8335660d13ce-kube-api-access-5gq65\") pod \"dca598ee-3551-4f28-b64c-8335660d13ce\" (UID: \"dca598ee-3551-4f28-b64c-8335660d13ce\") " Nov 04 01:45:16.884227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:16.884201 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca598ee-3551-4f28-b64c-8335660d13ce-maas-api-service-ca" pod "dca598ee-3551-4f28-b64c-8335660d13ce" (UID: "dca598ee-3551-4f28-b64c-8335660d13ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:16.886021 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:16.885996 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca598ee-3551-4f28-b64c-8335660d13ce-kube-api-access-5gq65" pod "dca598ee-3551-4f28-b64c-8335660d13ce" (UID: "dca598ee-3551-4f28-b64c-8335660d13ce"). InnerVolumeSpecName "kube-api-access-5gq65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:16.985251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:16.985186 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gq65\" (UniqueName: \"kubernetes.io/projected/dca598ee-3551-4f28-b64c-8335660d13ce-kube-api-access-5gq65\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:16.985251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:16.985212 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca598ee-3551-4f28-b64c-8335660d13ce-maas-api-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:17.650045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:17.650012 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" event={"ID":"dca598ee-3551-4f28-b64c-8335660d13ce","Type":"ContainerDied","Data":"3a2324ed68ca4f72bc7d517fefa81e65421f6e782672dd8db78ab3879afae377"} Nov 04 01:45:17.650234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:17.650059 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt" Nov 04 01:45:17.650234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:17.650074 4312 scope.go:122] "RemoveContainer" containerID="3e99bc93b9e360aa4b07f9116358443a5f0248450ba1fc80c2727ed7facfaba7" Nov 04 01:45:17.674533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:17.674506 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt"] Nov 04 01:45:17.678551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:17.678526 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-8h7dt"] Nov 04 01:45:17.687152 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:17.687126 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca598ee_3551_4f28_b64c_8335660d13ce.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:45:18.034163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034123 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm"] Nov 04 01:45:18.034786 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034767 4312 cpu_manager.go:414] "containerMap: removing container" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.034881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034788 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.034881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034806 4312 cpu_manager.go:414] "containerMap: removing container" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.034881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034815 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.034881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034831 4312 cpu_manager.go:414] "containerMap: removing container" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.034881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034841 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.035148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034974 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.035148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.034985 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.040443 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.040413 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.048892 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.048869 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:45:18.055450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.055428 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm"] Nov 04 01:45:18.074757 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.074698 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-dbe634-serving-cert\"" Nov 04 01:45:18.074914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.074796 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:45:18.074914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.074803 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rbkg2\"" Nov 04 01:45:18.094527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.094496 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.094731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.094646 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmgh\" (UniqueName: \"kubernetes.io/projected/f8139fac-4176-4d79-98a2-9975276c302a-kube-api-access-cfmgh\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.094800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.094766 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.094854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.094804 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8139fac-4176-4d79-98a2-9975276c302a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.102379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.100545 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:45:18.109661 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.109631 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" path="/var/lib/kubelet/pods/dca598ee-3551-4f28-b64c-8335660d13ce/volumes" Nov 04 01:45:18.195572 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.195541 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.195744 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.195634 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmgh\" (UniqueName: \"kubernetes.io/projected/f8139fac-4176-4d79-98a2-9975276c302a-kube-api-access-cfmgh\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.195744 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.195678 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.195744 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.195693 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8139fac-4176-4d79-98a2-9975276c302a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.195870 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:18.195813 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-serving-cert: secret "maas-api-e2e-worker-single-main-dbe634-serving-cert" not found Nov 04 01:45:18.195914 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:18.195894 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls podName:f8139fac-4176-4d79-98a2-9975276c302a nodeName:}" failed. No retries permitted until 2026-11-04 01:45:18.695870992 +0000 UTC m=+2550.718802977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls") pod "maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" (UID: "f8139fac-4176-4d79-98a2-9975276c302a") : secret "maas-api-e2e-worker-single-main-dbe634-serving-cert" not found Nov 04 01:45:18.196512 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.196484 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8139fac-4176-4d79-98a2-9975276c302a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.198392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.198367 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.208701 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.208675 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmgh\" (UniqueName: \"kubernetes.io/projected/f8139fac-4176-4d79-98a2-9975276c302a-kube-api-access-cfmgh\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.235055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.235033 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn"] Nov 04 01:45:18.235910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.235888 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca598ee-3551-4f28-b64c-8335660d13ce" containerName="cleanup" Nov 04 01:45:18.239174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.239156 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr"] Nov 04 01:45:18.239303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.239285 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.242582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.242564 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.253141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.253118 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dbe634\"" Nov 04 01:45:18.253634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.253542 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ec4a373b-dockercfg-mq8pp\"" Nov 04 01:45:18.260475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.260456 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr"] Nov 04 01:45:18.273436 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.265255 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn"] Nov 04 01:45:18.296714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.296622 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wfp\" (UniqueName: \"kubernetes.io/projected/7c94eae4-bae7-4f15-9497-22c130601245-kube-api-access-l2wfp\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.296873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.296830 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrb77\" (UniqueName: \"kubernetes.io/projected/0f5e3ab3-7269-4551-a2d4-251ffefe2134-kube-api-access-lrb77\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.296873 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.296860 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94eae4-bae7-4f15-9497-22c130601245-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.296995 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.296954 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f5e3ab3-7269-4551-a2d4-251ffefe2134-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.397837 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.397796 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f5e3ab3-7269-4551-a2d4-251ffefe2134-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.398041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.397875 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wfp\" (UniqueName: \"kubernetes.io/projected/7c94eae4-bae7-4f15-9497-22c130601245-kube-api-access-l2wfp\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.398041 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.398024 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrb77\" (UniqueName: \"kubernetes.io/projected/0f5e3ab3-7269-4551-a2d4-251ffefe2134-kube-api-access-lrb77\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.398160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.398070 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94eae4-bae7-4f15-9497-22c130601245-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.398800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.398774 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f5e3ab3-7269-4551-a2d4-251ffefe2134-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.398934 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.398912 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94eae4-bae7-4f15-9497-22c130601245-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.416224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.416192 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrb77\" (UniqueName: \"kubernetes.io/projected/0f5e3ab3-7269-4551-a2d4-251ffefe2134-kube-api-access-lrb77\") pod \"payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.416957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.416934 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wfp\" (UniqueName: \"kubernetes.io/projected/7c94eae4-bae7-4f15-9497-22c130601245-kube-api-access-l2wfp\") pod \"payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.554108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.554008 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:18.561344 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.561312 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:18.701225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.701190 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.704212 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.704183 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:18.836582 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.836541 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn"] Nov 04 01:45:18.836933 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:45:18.836907 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5e3ab3_7269_4551_a2d4_251ffefe2134.slice/crio-43b9a87fcf24911c6a6c416c7ec26b462e1e76245b90ca7834b8f46fb9ea5cd1 WatchSource:0}: Error finding container 43b9a87fcf24911c6a6c416c7ec26b462e1e76245b90ca7834b8f46fb9ea5cd1: Status 404 returned error can't find the container with id 43b9a87fcf24911c6a6c416c7ec26b462e1e76245b90ca7834b8f46fb9ea5cd1 Nov 04 01:45:18.861203 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.861179 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr"] Nov 04 01:45:18.869181 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:45:18.869146 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c94eae4_bae7_4f15_9497_22c130601245.slice/crio-6d6651c02a7689704cdd913b54a85bb06630817c28a5f222ca8aa9ac95e9bbbc WatchSource:0}: Error finding container 6d6651c02a7689704cdd913b54a85bb06630817c28a5f222ca8aa9ac95e9bbbc: Status 404 returned error can't find the container with id 6d6651c02a7689704cdd913b54a85bb06630817c28a5f222ca8aa9ac95e9bbbc Nov 04 01:45:18.958381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:18.958341 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:19.237694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.237666 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm"] Nov 04 01:45:19.238990 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:45:19.238961 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8139fac_4176_4d79_98a2_9975276c302a.slice/crio-ca320684001a94b2c3de1608bb9879b926cc8551034d0a3da51b51c4c72ae301 WatchSource:0}: Error finding container ca320684001a94b2c3de1608bb9879b926cc8551034d0a3da51b51c4c72ae301: Status 404 returned error can't find the container with id ca320684001a94b2c3de1608bb9879b926cc8551034d0a3da51b51c4c72ae301 Nov 04 01:45:19.664800 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.664758 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" event={"ID":"0f5e3ab3-7269-4551-a2d4-251ffefe2134","Type":"ContainerStarted","Data":"a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24"} Nov 04 01:45:19.664998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.664824 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" event={"ID":"0f5e3ab3-7269-4551-a2d4-251ffefe2134","Type":"ContainerStarted","Data":"43b9a87fcf24911c6a6c416c7ec26b462e1e76245b90ca7834b8f46fb9ea5cd1"} Nov 04 01:45:19.664998 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.664893 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:19.666653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.666625 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" event={"ID":"7c94eae4-bae7-4f15-9497-22c130601245","Type":"ContainerStarted","Data":"53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8"} Nov 04 01:45:19.666785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.666676 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" event={"ID":"7c94eae4-bae7-4f15-9497-22c130601245","Type":"ContainerStarted","Data":"6d6651c02a7689704cdd913b54a85bb06630817c28a5f222ca8aa9ac95e9bbbc"} Nov 04 01:45:19.666785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.666705 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:19.667999 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.667971 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" event={"ID":"f8139fac-4176-4d79-98a2-9975276c302a","Type":"ContainerStarted","Data":"ca320684001a94b2c3de1608bb9879b926cc8551034d0a3da51b51c4c72ae301"} Nov 04 01:45:19.704559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.704505 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" podStartSLOduration=1.70448682 podStartE2EDuration="1.70448682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:19.702986306 +0000 UTC m=+2551.725918314" watchObservedRunningTime="2026-11-04 01:45:19.70448682 +0000 UTC m=+2551.727418829" Nov 04 01:45:19.723280 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:19.723234 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" podStartSLOduration=1.723218165 podStartE2EDuration="1.723218165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:19.720836349 +0000 UTC m=+2551.743768380" watchObservedRunningTime="2026-11-04 01:45:19.723218165 +0000 UTC m=+2551.746150173" Nov 04 01:45:20.674531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:20.674493 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" event={"ID":"f8139fac-4176-4d79-98a2-9975276c302a","Type":"ContainerStarted","Data":"e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448"} Nov 04 01:45:20.675036 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:20.674735 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:20.704496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:20.704438 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" podStartSLOduration=2.263272327 podStartE2EDuration="2.70442067s" totalImagesPullingTime="441.148343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:20.704159731 +0000 UTC m=+2552.727091738" watchObservedRunningTime="2026-11-04 01:45:20.70442067 +0000 UTC m=+2552.727352679" Nov 04 01:45:26.685659 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:26.685628 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:29.464271 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:29.464179 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:45:30.677866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:30.677837 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:30.678320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:30.677920 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:37.264204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:37.264166 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn"] Nov 04 01:45:37.277694 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:37.277664 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr"] Nov 04 01:45:38.155068 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.155009 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm"] Nov 04 01:45:38.155409 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.155265 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" podUID="f8139fac-4176-4d79-98a2-9975276c302a" containerName="maas-api" containerID="cri-o://e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448" gracePeriod=30 Nov 04 01:45:38.441040 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.441011 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:38.531812 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.531782 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8139fac-4176-4d79-98a2-9975276c302a-kube-api-access-cfmgh\" (UniqueName: \"kubernetes.io/projected/f8139fac-4176-4d79-98a2-9975276c302a-kube-api-access-cfmgh\") pod \"f8139fac-4176-4d79-98a2-9975276c302a\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " Nov 04 01:45:38.532076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.531830 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls\") pod \"f8139fac-4176-4d79-98a2-9975276c302a\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " Nov 04 01:45:38.532076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.531886 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8139fac-4176-4d79-98a2-9975276c302a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8139fac-4176-4d79-98a2-9975276c302a-openshift-service-ca\") pod \"f8139fac-4176-4d79-98a2-9975276c302a\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " Nov 04 01:45:38.532076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.531944 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-metrics-certs\") pod \"f8139fac-4176-4d79-98a2-9975276c302a\" (UID: \"f8139fac-4176-4d79-98a2-9975276c302a\") " Nov 04 01:45:38.532325 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.532296 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8139fac-4176-4d79-98a2-9975276c302a-openshift-service-ca" pod "f8139fac-4176-4d79-98a2-9975276c302a" (UID: "f8139fac-4176-4d79-98a2-9975276c302a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:38.533965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.533930 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls" pod "f8139fac-4176-4d79-98a2-9975276c302a" (UID: "f8139fac-4176-4d79-98a2-9975276c302a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:45:38.533965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.533944 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8139fac-4176-4d79-98a2-9975276c302a-kube-api-access-cfmgh" pod "f8139fac-4176-4d79-98a2-9975276c302a" (UID: "f8139fac-4176-4d79-98a2-9975276c302a"). InnerVolumeSpecName "kube-api-access-cfmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:38.533965 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.533948 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-metrics-certs" pod "f8139fac-4176-4d79-98a2-9975276c302a" (UID: "f8139fac-4176-4d79-98a2-9975276c302a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:45:38.632805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.632768 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfmgh\" (UniqueName: \"kubernetes.io/projected/f8139fac-4176-4d79-98a2-9975276c302a-kube-api-access-cfmgh\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:38.632805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.632799 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:38.632805 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.632811 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8139fac-4176-4d79-98a2-9975276c302a-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:38.633045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.632819 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8139fac-4176-4d79-98a2-9975276c302a-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:38.789500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.789465 4312 generic.go:356] "Generic (PLEG): container finished" podID="f8139fac-4176-4d79-98a2-9975276c302a" containerID="e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448" exitCode=0 Nov 04 01:45:38.789678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.789514 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" Nov 04 01:45:38.789678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.789517 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" event={"ID":"f8139fac-4176-4d79-98a2-9975276c302a","Type":"ContainerDied","Data":"e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448"} Nov 04 01:45:38.789678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.789634 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm" event={"ID":"f8139fac-4176-4d79-98a2-9975276c302a","Type":"ContainerDied","Data":"ca320684001a94b2c3de1608bb9879b926cc8551034d0a3da51b51c4c72ae301"} Nov 04 01:45:38.789678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.789657 4312 scope.go:122] "RemoveContainer" containerID="e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448" Nov 04 01:45:38.799780 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.799758 4312 scope.go:122] "RemoveContainer" containerID="e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448" Nov 04 01:45:38.800066 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:38.800044 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448\": container with ID starting with e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448 not found: ID does not exist" containerID="e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448" Nov 04 01:45:38.800178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.800075 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448"} err="failed to get container status \"e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448\": rpc error: code = NotFound desc = could not find container \"e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448\": container with ID starting with e5f4df62772f92a6df4841eae6f06645d62088958a28ceec07e8320666070448 not found: ID does not exist" Nov 04 01:45:38.814030 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.814004 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm"] Nov 04 01:45:38.816263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.816239 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe634-6d497c9646-h5wqm"] Nov 04 01:45:38.865230 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.865198 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b"] Nov 04 01:45:38.865506 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:38.865479 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" containerID="cri-o://aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e" gracePeriod=30 Nov 04 01:45:40.108726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:40.108690 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8139fac-4176-4d79-98a2-9975276c302a" path="/var/lib/kubelet/pods/f8139fac-4176-4d79-98a2-9975276c302a/volumes" Nov 04 01:45:42.264853 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.264810 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" podUID="0f5e3ab3-7269-4551-a2d4-251ffefe2134" containerName="payload-processing" containerID="cri-o://a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24" gracePeriod=25 Nov 04 01:45:42.278163 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.278091 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" podUID="7c94eae4-bae7-4f15-9497-22c130601245" containerName="payload-pre-processing" containerID="cri-o://53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8" gracePeriod=25 Nov 04 01:45:42.553830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.553806 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:42.576128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.576101 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:42.691032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.690997 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c94eae4-bae7-4f15-9497-22c130601245-kube-api-access-l2wfp\" (UniqueName: \"kubernetes.io/projected/7c94eae4-bae7-4f15-9497-22c130601245-kube-api-access-l2wfp\") pod \"7c94eae4-bae7-4f15-9497-22c130601245\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " Nov 04 01:45:42.691215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.691153 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5e3ab3-7269-4551-a2d4-251ffefe2134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f5e3ab3-7269-4551-a2d4-251ffefe2134-plugins-config-volume\") pod \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " Nov 04 01:45:42.691215 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.691194 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c94eae4-bae7-4f15-9497-22c130601245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94eae4-bae7-4f15-9497-22c130601245-plugins-config-volume\") pod \"7c94eae4-bae7-4f15-9497-22c130601245\" (UID: \"7c94eae4-bae7-4f15-9497-22c130601245\") " Nov 04 01:45:42.691345 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.691243 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5e3ab3-7269-4551-a2d4-251ffefe2134-kube-api-access-lrb77\" (UniqueName: \"kubernetes.io/projected/0f5e3ab3-7269-4551-a2d4-251ffefe2134-kube-api-access-lrb77\") pod \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\" (UID: \"0f5e3ab3-7269-4551-a2d4-251ffefe2134\") " Nov 04 01:45:42.691655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.691602 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c94eae4-bae7-4f15-9497-22c130601245-plugins-config-volume" pod "7c94eae4-bae7-4f15-9497-22c130601245" (UID: "7c94eae4-bae7-4f15-9497-22c130601245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:42.691655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.691631 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5e3ab3-7269-4551-a2d4-251ffefe2134-plugins-config-volume" pod "0f5e3ab3-7269-4551-a2d4-251ffefe2134" (UID: "0f5e3ab3-7269-4551-a2d4-251ffefe2134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:42.693211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.693186 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c94eae4-bae7-4f15-9497-22c130601245-kube-api-access-l2wfp" pod "7c94eae4-bae7-4f15-9497-22c130601245" (UID: "7c94eae4-bae7-4f15-9497-22c130601245"). InnerVolumeSpecName "kube-api-access-l2wfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:42.693329 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.693314 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5e3ab3-7269-4551-a2d4-251ffefe2134-kube-api-access-lrb77" pod "0f5e3ab3-7269-4551-a2d4-251ffefe2134" (UID: "0f5e3ab3-7269-4551-a2d4-251ffefe2134"). InnerVolumeSpecName "kube-api-access-lrb77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:42.792559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.792485 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f5e3ab3-7269-4551-a2d4-251ffefe2134-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:42.792559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.792513 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c94eae4-bae7-4f15-9497-22c130601245-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:42.792559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.792526 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrb77\" (UniqueName: \"kubernetes.io/projected/0f5e3ab3-7269-4551-a2d4-251ffefe2134-kube-api-access-lrb77\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:42.792559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.792534 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wfp\" (UniqueName: \"kubernetes.io/projected/7c94eae4-bae7-4f15-9497-22c130601245-kube-api-access-l2wfp\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:42.813058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.813030 4312 generic.go:356] "Generic (PLEG): container finished" podID="0f5e3ab3-7269-4551-a2d4-251ffefe2134" containerID="a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24" exitCode=0 Nov 04 01:45:42.813224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.813086 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" Nov 04 01:45:42.813224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.813112 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" event={"ID":"0f5e3ab3-7269-4551-a2d4-251ffefe2134","Type":"ContainerDied","Data":"a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24"} Nov 04 01:45:42.813224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.813188 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn" event={"ID":"0f5e3ab3-7269-4551-a2d4-251ffefe2134","Type":"ContainerDied","Data":"43b9a87fcf24911c6a6c416c7ec26b462e1e76245b90ca7834b8f46fb9ea5cd1"} Nov 04 01:45:42.813224 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.813212 4312 scope.go:122] "RemoveContainer" containerID="a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24" Nov 04 01:45:42.814554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.814531 4312 generic.go:356] "Generic (PLEG): container finished" podID="7c94eae4-bae7-4f15-9497-22c130601245" containerID="53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8" exitCode=0 Nov 04 01:45:42.814674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.814573 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" Nov 04 01:45:42.814674 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.814616 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" event={"ID":"7c94eae4-bae7-4f15-9497-22c130601245","Type":"ContainerDied","Data":"53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8"} Nov 04 01:45:42.814827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.814674 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr" event={"ID":"7c94eae4-bae7-4f15-9497-22c130601245","Type":"ContainerDied","Data":"6d6651c02a7689704cdd913b54a85bb06630817c28a5f222ca8aa9ac95e9bbbc"} Nov 04 01:45:42.823051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.823027 4312 scope.go:122] "RemoveContainer" containerID="a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24" Nov 04 01:45:42.823322 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:42.823301 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24\": container with ID starting with a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24 not found: ID does not exist" containerID="a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24" Nov 04 01:45:42.823441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.823358 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24"} err="failed to get container status \"a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24\": rpc error: code = NotFound desc = could not find container \"a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24\": container with ID starting with a1d0f048c2d88a9768b30e496c7e1e5d48cf272fac5aafbbfa789a1ecc17aa24 not found: ID does not exist" Nov 04 01:45:42.823441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.823396 4312 scope.go:122] "RemoveContainer" containerID="53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8" Nov 04 01:45:42.832411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.832328 4312 scope.go:122] "RemoveContainer" containerID="53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8" Nov 04 01:45:42.832674 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:42.832654 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8\": container with ID starting with 53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8 not found: ID does not exist" containerID="53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8" Nov 04 01:45:42.832768 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.832679 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8"} err="failed to get container status \"53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8\": rpc error: code = NotFound desc = could not find container \"53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8\": container with ID starting with 53aa6d327700c1bb092ac82b15558a8d0836d8bc4d3d447ee383c253290173c8 not found: ID does not exist" Nov 04 01:45:42.839354 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.839314 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr"] Nov 04 01:45:42.843431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.843408 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dbe634-7f8b6ssccr"] Nov 04 01:45:42.854971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.854947 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn"] Nov 04 01:45:42.861073 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:42.861053 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe634-78d7dfc4dlqfmn"] Nov 04 01:45:43.619301 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.619213 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m"] Nov 04 01:45:43.619887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.619865 4312 cpu_manager.go:414] "containerMap: removing container" podUID="7c94eae4-bae7-4f15-9497-22c130601245" containerName="payload-pre-processing" Nov 04 01:45:43.619981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.619891 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c94eae4-bae7-4f15-9497-22c130601245" containerName="payload-pre-processing" Nov 04 01:45:43.619981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.619907 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f8139fac-4176-4d79-98a2-9975276c302a" containerName="maas-api" Nov 04 01:45:43.619981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.619915 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8139fac-4176-4d79-98a2-9975276c302a" containerName="maas-api" Nov 04 01:45:43.619981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.619966 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0f5e3ab3-7269-4551-a2d4-251ffefe2134" containerName="payload-processing" Nov 04 01:45:43.619981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.619975 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5e3ab3-7269-4551-a2d4-251ffefe2134" containerName="payload-processing" Nov 04 01:45:43.620243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.620082 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c94eae4-bae7-4f15-9497-22c130601245" containerName="payload-pre-processing" Nov 04 01:45:43.620243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.620094 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8139fac-4176-4d79-98a2-9975276c302a" containerName="maas-api" Nov 04 01:45:43.620243 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.620111 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5e3ab3-7269-4551-a2d4-251ffefe2134" containerName="payload-processing" Nov 04 01:45:43.624685 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.624660 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.633008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.632982 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f3467868-kserve-self-signed-certs\"" Nov 04 01:45:43.636142 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.636120 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m"] Nov 04 01:45:43.802769 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.802732 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnl2k\" (UniqueName: \"kubernetes.io/projected/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-kube-api-access-lnl2k\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.802974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.802794 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-home\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.802974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.802918 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-dshm\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.802974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.802966 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tls-certs\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.803191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.802991 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tmp-dir\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.803191 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.803105 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-model-cache\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904378 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904328 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnl2k\" (UniqueName: \"kubernetes.io/projected/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-kube-api-access-lnl2k\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904404 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-home\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904438 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-dshm\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904473 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tls-certs\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904493 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tmp-dir\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904678 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904543 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-model-cache\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904847 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904820 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-home\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904879 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-model-cache\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.904955 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.904907 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tmp-dir\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.906691 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.906666 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-dshm\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.906914 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.906899 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tls-certs\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.916593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.916575 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnl2k\" (UniqueName: \"kubernetes.io/projected/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-kube-api-access-lnl2k\") pod \"e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:43.939577 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:43.939553 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:44.109026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.108996 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5e3ab3-7269-4551-a2d4-251ffefe2134" path="/var/lib/kubelet/pods/0f5e3ab3-7269-4551-a2d4-251ffefe2134/volumes" Nov 04 01:45:44.109450 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.109434 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c94eae4-bae7-4f15-9497-22c130601245" path="/var/lib/kubelet/pods/7c94eae4-bae7-4f15-9497-22c130601245/volumes" Nov 04 01:45:44.146179 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.146124 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:45:44.215014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.214973 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-envoy\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.217374 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.217158 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-envoy" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:45:44.217480 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.217460 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m"] Nov 04 01:45:44.219725 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:45:44.219703 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod103fab8e_782e_4c86_8c8f_d5eecbeb74d1.slice/crio-4e1a21151e1d15e89978ac3edcfad4c3e667e6328835b6996981853cd5df7de2 WatchSource:0}: Error finding container 4e1a21151e1d15e89978ac3edcfad4c3e667e6328835b6996981853cd5df7de2: Status 404 returned error can't find the container with id 4e1a21151e1d15e89978ac3edcfad4c3e667e6328835b6996981853cd5df7de2 Nov 04 01:45:44.315705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315674 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-socket\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.315881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315732 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-istio-token\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-istio-token\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.315881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315768 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-kube-api-access-p62k9\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-kube-api-access-p62k9\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.315881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315803 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-certs\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.315881 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315834 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8703487-3195-44d9-9252-c07daf360c1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8703487-3195-44d9-9252-c07daf360c1a-istiod-ca-cert\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.316129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315900 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-credential-socket\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.316129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315937 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-socket" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:45:44.316129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.315949 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8703487-3195-44d9-9252-c07daf360c1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8703487-3195-44d9-9252-c07daf360c1a-istio-podinfo\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.316129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316050 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-data\") pod \"a8703487-3195-44d9-9252-c07daf360c1a\" (UID: \"a8703487-3195-44d9-9252-c07daf360c1a\") " Nov 04 01:45:44.316129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316092 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-certs" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:45:44.316439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316186 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8703487-3195-44d9-9252-c07daf360c1a-istiod-ca-cert" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:44.316439 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316281 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-credential-socket" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:45:44.316553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316452 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.316553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316472 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.316553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316487 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.316553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316500 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8703487-3195-44d9-9252-c07daf360c1a-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.316553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316514 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.316825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.316544 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-data" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:45:44.318505 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.318471 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-kube-api-access-p62k9" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "kube-api-access-p62k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:44.318680 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.318660 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-istio-token" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:44.318846 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.318823 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8703487-3195-44d9-9252-c07daf360c1a-istio-podinfo" pod "a8703487-3195-44d9-9252-c07daf360c1a" (UID: "a8703487-3195-44d9-9252-c07daf360c1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:45:44.417162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.417080 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.417162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.417109 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62k9\" (UniqueName: \"kubernetes.io/projected/a8703487-3195-44d9-9252-c07daf360c1a-kube-api-access-p62k9\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.417162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.417119 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8703487-3195-44d9-9252-c07daf360c1a-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.417162 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.417129 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8703487-3195-44d9-9252-c07daf360c1a-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:44.833482 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.833449 4312 generic.go:356] "Generic (PLEG): container finished" podID="a8703487-3195-44d9-9252-c07daf360c1a" containerID="aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e" exitCode=0 Nov 04 01:45:44.833940 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.833505 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" Nov 04 01:45:44.833940 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.833539 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" event={"ID":"a8703487-3195-44d9-9252-c07daf360c1a","Type":"ContainerDied","Data":"aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e"} Nov 04 01:45:44.833940 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.833592 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b" event={"ID":"a8703487-3195-44d9-9252-c07daf360c1a","Type":"ContainerDied","Data":"8411da1f205efbda92aaadad94fd22cd4bd03052abae878e7b5bb212ad550895"} Nov 04 01:45:44.833940 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.833620 4312 scope.go:122] "RemoveContainer" containerID="aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e" Nov 04 01:45:44.835583 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.835558 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" event={"ID":"103fab8e-782e-4c86-8c8f-d5eecbeb74d1","Type":"ContainerStarted","Data":"2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069"} Nov 04 01:45:44.835715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.835610 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" event={"ID":"103fab8e-782e-4c86-8c8f-d5eecbeb74d1","Type":"ContainerStarted","Data":"4e1a21151e1d15e89978ac3edcfad4c3e667e6328835b6996981853cd5df7de2"} Nov 04 01:45:44.844009 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.843983 4312 scope.go:122] "RemoveContainer" containerID="b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc" Nov 04 01:45:44.852227 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.852210 4312 scope.go:122] "RemoveContainer" containerID="aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e" Nov 04 01:45:44.852494 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:44.852477 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e\": container with ID starting with aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e not found: ID does not exist" containerID="aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e" Nov 04 01:45:44.852542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.852501 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e"} err="failed to get container status \"aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e\": rpc error: code = NotFound desc = could not find container \"aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e\": container with ID starting with aba6cc7df5c2dee8c4553caf30d34e9d5cb5c5e5158a2a8700bc4dd060b35b0e not found: ID does not exist" Nov 04 01:45:44.852542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.852516 4312 scope.go:122] "RemoveContainer" containerID="b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc" Nov 04 01:45:44.852777 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:44.852749 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc\": container with ID starting with b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc not found: ID does not exist" containerID="b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc" Nov 04 01:45:44.852821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.852788 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc"} err="failed to get container status \"b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc\": rpc error: code = NotFound desc = could not find container \"b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc\": container with ID starting with b8706449f70fa838a46b94faa5807aaec01279304bad5933163ebd86d7c2c1cc not found: ID does not exist" Nov 04 01:45:44.858256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.858208 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" podStartSLOduration=1.858195357 podStartE2EDuration="1.858195357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:44.856444953 +0000 UTC m=+2576.879376985" watchObservedRunningTime="2026-11-04 01:45:44.858195357 +0000 UTC m=+2576.881127364" Nov 04 01:45:44.870852 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.870830 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b"] Nov 04 01:45:44.877108 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.877088 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe634-openshift-default-69684858cck227b"] Nov 04 01:45:44.940606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:44.940582 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:46.118423 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:46.118315 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8703487-3195-44d9-9252-c07daf360c1a" path="/var/lib/kubelet/pods/a8703487-3195-44d9-9252-c07daf360c1a/volumes" Nov 04 01:45:50.896498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:50.896465 4312 scope.go:122] "RemoveContainer" containerID="c09a6457b63eaa2b12f4df1cc60962c90e4a130821eb64294066995e154b1c39" Nov 04 01:45:53.939699 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:53.939662 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:53.952511 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:53.952488 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:54.942854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:45:54.942823 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:45:59.457384 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:45:59.457329 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:46:00.137546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.137516 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l"] Nov 04 01:46:00.138003 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.137985 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" Nov 04 01:46:00.138098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.138005 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" Nov 04 01:46:00.138098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.138030 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" Nov 04 01:46:00.138098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.138038 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" Nov 04 01:46:00.138254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.138202 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" Nov 04 01:46:00.138254 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.138220 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8703487-3195-44d9-9252-c07daf360c1a" containerName="istio-proxy" Nov 04 01:46:00.143660 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.143630 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.148878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.148848 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f3467868-kserve-self-signed-certs\"" Nov 04 01:46:00.153204 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.153178 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l"] Nov 04 01:46:00.239697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.239655 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-dshm\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.239870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.239709 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-model-cache\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.239870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.239807 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tls-certs\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.239870 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.239836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tmp-dir\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.240045 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.240025 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-home\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.240115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.240069 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r786f\" (UniqueName: \"kubernetes.io/projected/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-kube-api-access-r786f\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341431 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-home\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341481 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r786f\" (UniqueName: \"kubernetes.io/projected/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-kube-api-access-r786f\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341511 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-dshm\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341534 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-model-cache\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341563 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tls-certs\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341579 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tmp-dir\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341885 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-home\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.341964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341940 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-model-cache\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.342046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.341975 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tmp-dir\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.343926 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.343893 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-dshm\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.344210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.344189 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tls-certs\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.354431 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.354410 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r786f\" (UniqueName: \"kubernetes.io/projected/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-kube-api-access-r786f\") pod \"e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.462388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.462310 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:00.740717 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.740690 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l"] Nov 04 01:46:00.742025 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:46:00.741993 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda72bce0f_3a71_44c7_ba8a_0db048c7d5ea.slice/crio-cc8e5f42cc728eee96846237748e26ba53891c9a9c4f9fb236e7e2dc08f10bb4 WatchSource:0}: Error finding container cc8e5f42cc728eee96846237748e26ba53891c9a9c4f9fb236e7e2dc08f10bb4: Status 404 returned error can't find the container with id cc8e5f42cc728eee96846237748e26ba53891c9a9c4f9fb236e7e2dc08f10bb4 Nov 04 01:46:00.962285 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.962253 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" event={"ID":"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea","Type":"ContainerStarted","Data":"ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a"} Nov 04 01:46:00.962467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.962294 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" event={"ID":"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea","Type":"ContainerStarted","Data":"cc8e5f42cc728eee96846237748e26ba53891c9a9c4f9fb236e7e2dc08f10bb4"} Nov 04 01:46:00.985176 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:00.985119 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" podStartSLOduration=0.985102403 podStartE2EDuration="985.102403ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:00.983751102 +0000 UTC m=+2593.006683111" watchObservedRunningTime="2026-11-04 01:46:00.985102403 +0000 UTC m=+2593.008034410" Nov 04 01:46:01.462503 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:01.462464 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:10.463018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:10.462982 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:10.475644 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:10.475614 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:11.052496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:11.052464 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:16.613445 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.613407 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn"] Nov 04 01:46:16.617053 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.617033 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.624360 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.624318 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:46:16.625820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.625167 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:46:16.625820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.625220 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:46:16.626014 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.625260 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:46:16.627470 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.627448 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn"] Nov 04 01:46:16.669709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.669676 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.669856 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.669741 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.669905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.669886 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzzc\" (UniqueName: \"kubernetes.io/projected/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-kube-api-access-lxzzc\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.771272 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.771235 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.771458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.771288 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.771458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.771363 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzzc\" (UniqueName: \"kubernetes.io/projected/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-kube-api-access-lxzzc\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.772039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.772016 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.773705 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.773686 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.783794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.783766 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzzc\" (UniqueName: \"kubernetes.io/projected/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-kube-api-access-lxzzc\") pod \"maas-api-revoke-sub-452b6d7b-2g4kn\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:16.933043 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:16.932957 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:17.213742 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:17.213715 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn"] Nov 04 01:46:17.215055 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:46:17.215031 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023a276e_0c6e_4b23_9d79_25c5bfc8af3b.slice/crio-d5b5f1b337b63555cd06bd3a027f6b85baa8e017f1d3afe6cd5b7f3331681b8c WatchSource:0}: Error finding container d5b5f1b337b63555cd06bd3a027f6b85baa8e017f1d3afe6cd5b7f3331681b8c: Status 404 returned error can't find the container with id d5b5f1b337b63555cd06bd3a027f6b85baa8e017f1d3afe6cd5b7f3331681b8c Nov 04 01:46:18.081095 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:18.081055 4312 generic.go:356] "Generic (PLEG): container finished" podID="023a276e-0c6e-4b23-9d79-25c5bfc8af3b" containerID="1b67081abfd559368c84543f37807e82acc4453f83ef4166504f40aa11510267" exitCode=0 Nov 04 01:46:18.081698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:18.081473 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" event={"ID":"023a276e-0c6e-4b23-9d79-25c5bfc8af3b","Type":"ContainerDied","Data":"1b67081abfd559368c84543f37807e82acc4453f83ef4166504f40aa11510267"} Nov 04 01:46:18.081791 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:18.081775 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" event={"ID":"023a276e-0c6e-4b23-9d79-25c5bfc8af3b","Type":"ContainerStarted","Data":"d5b5f1b337b63555cd06bd3a027f6b85baa8e017f1d3afe6cd5b7f3331681b8c"} Nov 04 01:46:19.673816 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.673502 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:19.769775 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.769728 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-xks-ca\") pod \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " Nov 04 01:46:19.777355 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.773546 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-openshift-ca\") pod \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " Nov 04 01:46:19.800451 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.799742 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-kube-api-access-lxzzc\" (UniqueName: \"kubernetes.io/projected/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-kube-api-access-lxzzc\") pod \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\" (UID: \"023a276e-0c6e-4b23-9d79-25c5bfc8af3b\") " Nov 04 01:46:19.801234 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.801207 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-openshift-ca" pod "023a276e-0c6e-4b23-9d79-25c5bfc8af3b" (UID: "023a276e-0c6e-4b23-9d79-25c5bfc8af3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:19.801722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.801703 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:19.858903 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.858843 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-kube-api-access-lxzzc" pod "023a276e-0c6e-4b23-9d79-25c5bfc8af3b" (UID: "023a276e-0c6e-4b23-9d79-25c5bfc8af3b"). InnerVolumeSpecName "kube-api-access-lxzzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:19.867553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.867487 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-xks-ca" pod "023a276e-0c6e-4b23-9d79-25c5bfc8af3b" (UID: "023a276e-0c6e-4b23-9d79-25c5bfc8af3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:19.932367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.928995 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxzzc\" (UniqueName: \"kubernetes.io/projected/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-kube-api-access-lxzzc\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:19.932367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:19.929033 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023a276e-0c6e-4b23-9d79-25c5bfc8af3b-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:20.143402 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:20.131774 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" Nov 04 01:46:20.180358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:20.179615 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn" event={"ID":"023a276e-0c6e-4b23-9d79-25c5bfc8af3b","Type":"ContainerDied","Data":"d5b5f1b337b63555cd06bd3a027f6b85baa8e017f1d3afe6cd5b7f3331681b8c"} Nov 04 01:46:20.180358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:20.179691 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5b5f1b337b63555cd06bd3a027f6b85baa8e017f1d3afe6cd5b7f3331681b8c" Nov 04 01:46:27.473676 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:27.473625 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m"] Nov 04 01:46:28.635798 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:28.635402 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l"] Nov 04 01:46:29.024765 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:29.024677 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594"] Nov 04 01:46:29.036761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:29.036737 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gz594"] Nov 04 01:46:29.463265 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:46:29.463227 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:46:30.142978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.142941 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9985080-b9d5-4c7a-925d-24982dcb4f0e" path="/var/lib/kubelet/pods/b9985080-b9d5-4c7a-925d-24982dcb4f0e/volumes" Nov 04 01:46:30.220067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.220029 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp"] Nov 04 01:46:30.220709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.220685 4312 cpu_manager.go:414] "containerMap: removing container" podUID="023a276e-0c6e-4b23-9d79-25c5bfc8af3b" containerName="revoke-keys" Nov 04 01:46:30.220853 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.220710 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023a276e-0c6e-4b23-9d79-25c5bfc8af3b" containerName="revoke-keys" Nov 04 01:46:30.220941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.220868 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="023a276e-0c6e-4b23-9d79-25c5bfc8af3b" containerName="revoke-keys" Nov 04 01:46:30.224971 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.224946 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.271359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.245809 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:46:30.271359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.246522 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:46:30.271359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.246654 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:46:30.271359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.246699 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:46:30.271359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.250719 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp"] Nov 04 01:46:30.276368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.275042 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.276368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.275101 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zlv8\" (UniqueName: \"kubernetes.io/projected/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-kube-api-access-6zlv8\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.276368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.275213 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.375936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.375896 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.376225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.376201 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zlv8\" (UniqueName: \"kubernetes.io/projected/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-kube-api-access-6zlv8\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.376496 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.376474 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.378947 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.378921 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.379734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.379711 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.391812 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.391781 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zlv8\" (UniqueName: \"kubernetes.io/projected/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-kube-api-access-6zlv8\") pod \"maas-api-revoke-sub-3e31a506-qr9rp\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.552911 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.552868 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:30.905052 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:30.905010 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp"] Nov 04 01:46:30.911634 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:46:30.907606 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786f49ba_6bb9_441d_9c8a_8f2d5e8f3299.slice/crio-7ecec11db05af89f234a4c156549149d917b2bcac2b6ce5d2bf8a8daa778d90b WatchSource:0}: Error finding container 7ecec11db05af89f234a4c156549149d917b2bcac2b6ce5d2bf8a8daa778d90b: Status 404 returned error can't find the container with id 7ecec11db05af89f234a4c156549149d917b2bcac2b6ce5d2bf8a8daa778d90b Nov 04 01:46:31.280368 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:31.272321 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" event={"ID":"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299","Type":"ContainerStarted","Data":"7ecec11db05af89f234a4c156549149d917b2bcac2b6ce5d2bf8a8daa778d90b"} Nov 04 01:46:32.303372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:32.303311 4312 generic.go:356] "Generic (PLEG): container finished" podID="786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" containerID="e2ea5e54ef3096cf750dede550e2f3632e074f47ce993f68c4cc1051c6650325" exitCode=0 Nov 04 01:46:32.304166 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:32.304122 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" event={"ID":"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299","Type":"ContainerDied","Data":"e2ea5e54ef3096cf750dede550e2f3632e074f47ce993f68c4cc1051c6650325"} Nov 04 01:46:32.516416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:32.515485 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:46:32.516416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:32.515587 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:46:33.680902 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.680871 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:33.797568 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.797533 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-kube-api-access-6zlv8\" (UniqueName: \"kubernetes.io/projected/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-kube-api-access-6zlv8\") pod \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " Nov 04 01:46:33.797854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.797622 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-xks-ca\") pod \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " Nov 04 01:46:33.797854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.797721 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-openshift-ca\") pod \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\" (UID: \"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299\") " Nov 04 01:46:33.798533 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.798474 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-openshift-ca" pod "786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" (UID: "786f49ba-6bb9-441d-9c8a-8f2d5e8f3299"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:33.800673 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.800646 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-xks-ca" pod "786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" (UID: "786f49ba-6bb9-441d-9c8a-8f2d5e8f3299"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:33.801055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.801033 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-kube-api-access-6zlv8" pod "786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" (UID: "786f49ba-6bb9-441d-9c8a-8f2d5e8f3299"). InnerVolumeSpecName "kube-api-access-6zlv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:33.899257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.899164 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zlv8\" (UniqueName: \"kubernetes.io/projected/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-kube-api-access-6zlv8\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:33.899257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.899206 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:33.899257 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:33.899223 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:34.327027 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:34.326985 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" event={"ID":"786f49ba-6bb9-441d-9c8a-8f2d5e8f3299","Type":"ContainerDied","Data":"7ecec11db05af89f234a4c156549149d917b2bcac2b6ce5d2bf8a8daa778d90b"} Nov 04 01:46:34.327259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:34.327063 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ecec11db05af89f234a4c156549149d917b2bcac2b6ce5d2bf8a8daa778d90b" Nov 04 01:46:34.327259 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:34.327145 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp" Nov 04 01:46:42.501510 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:42.501424 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" podUID="103fab8e-782e-4c86-8c8f-d5eecbeb74d1" containerName="main" containerID="cri-o://2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069" gracePeriod=45 Nov 04 01:46:43.331358 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.328680 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:46:43.458425 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.432973 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tls-certs\") pod \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " Nov 04 01:46:43.459588 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.459549 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-home\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-home\") pod \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.464455 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-home" pod "103fab8e-782e-4c86-8c8f-d5eecbeb74d1" (UID: "103fab8e-782e-4c86-8c8f-d5eecbeb74d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.464564 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-model-cache\") pod \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.464931 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-model-cache" pod "103fab8e-782e-4c86-8c8f-d5eecbeb74d1" (UID: "103fab8e-782e-4c86-8c8f-d5eecbeb74d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.465027 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-dshm\") pod \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.465672 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-kube-api-access-lnl2k\" (UniqueName: \"kubernetes.io/projected/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-kube-api-access-lnl2k\") pod \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.465743 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tmp-dir\") pod \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\" (UID: \"103fab8e-782e-4c86-8c8f-d5eecbeb74d1\") " Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.466162 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:43.470281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.466185 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:43.474165 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.473802 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tmp-dir" pod "103fab8e-782e-4c86-8c8f-d5eecbeb74d1" (UID: "103fab8e-782e-4c86-8c8f-d5eecbeb74d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.530326 4312 generic.go:356] "Generic (PLEG): container finished" podID="103fab8e-782e-4c86-8c8f-d5eecbeb74d1" containerID="2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069" exitCode=0 Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.530525 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" event={"ID":"103fab8e-782e-4c86-8c8f-d5eecbeb74d1","Type":"ContainerDied","Data":"2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069"} Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.530618 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" event={"ID":"103fab8e-782e-4c86-8c8f-d5eecbeb74d1","Type":"ContainerDied","Data":"4e1a21151e1d15e89978ac3edcfad4c3e667e6328835b6996981853cd5df7de2"} Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.530651 4312 scope.go:122] "RemoveContainer" containerID="2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069" Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.530863 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.534935 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tls-certs" pod "103fab8e-782e-4c86-8c8f-d5eecbeb74d1" (UID: "103fab8e-782e-4c86-8c8f-d5eecbeb74d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.542456 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-dshm" pod "103fab8e-782e-4c86-8c8f-d5eecbeb74d1" (UID: "103fab8e-782e-4c86-8c8f-d5eecbeb74d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:43.546359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.542596 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-kube-api-access-lnl2k" pod "103fab8e-782e-4c86-8c8f-d5eecbeb74d1" (UID: "103fab8e-782e-4c86-8c8f-d5eecbeb74d1"). InnerVolumeSpecName "kube-api-access-lnl2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:43.554521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.552926 4312 scope.go:122] "RemoveContainer" containerID="2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069" Nov 04 01:46:43.554521 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:46:43.553532 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069\": container with ID starting with 2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069 not found: ID does not exist" containerID="2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069" Nov 04 01:46:43.554521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.553604 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069"} err="failed to get container status \"2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069\": rpc error: code = NotFound desc = could not find container \"2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069\": container with ID starting with 2f9fbea465d0469da273b3ca0cf38b9356124cf695caa050eeb0c8b17a7bc069 not found: ID does not exist" Nov 04 01:46:43.567010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.566881 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:43.567010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.566916 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnl2k\" (UniqueName: \"kubernetes.io/projected/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-kube-api-access-lnl2k\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:43.567010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.566960 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:43.567010 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.566977 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103fab8e-782e-4c86-8c8f-d5eecbeb74d1-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:43.688397 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.679560 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" podUID="a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" containerName="main" containerID="cri-o://ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a" gracePeriod=45 Nov 04 01:46:43.919107 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.919038 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m"] Nov 04 01:46:43.923574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.923529 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m"] Nov 04 01:46:43.975825 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:43.975169 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-f3467868-kserve-6ff68659d9-6x26m" podUID="103fab8e-782e-4c86-8c8f-d5eecbeb74d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.221:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:46:44.129466 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.129015 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103fab8e-782e-4c86-8c8f-d5eecbeb74d1" path="/var/lib/kubelet/pods/103fab8e-782e-4c86-8c8f-d5eecbeb74d1/volumes" Nov 04 01:46:44.131195 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.131163 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.191426 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-home\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-home\") pod \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.191520 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tmp-dir\") pod \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.191582 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-model-cache\") pod \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.191678 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-dshm\") pod \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.191754 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-kube-api-access-r786f\" (UniqueName: \"kubernetes.io/projected/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-kube-api-access-r786f\") pod \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.191788 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tls-certs\") pod \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\" (UID: \"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea\") " Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.191831 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-model-cache" pod "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" (UID: "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:44.192353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.192163 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-model-cache\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:44.192919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.192414 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tmp-dir" pod "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" (UID: "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:44.192919 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.192557 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-home" pod "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" (UID: "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:44.195157 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.195130 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tls-certs" pod "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" (UID: "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:44.198970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.198941 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-kube-api-access-r786f" pod "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" (UID: "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea"). InnerVolumeSpecName "kube-api-access-r786f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:44.200018 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.199995 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-dshm" pod "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" (UID: "a72bce0f-3a71-44c7-ba8a-0db048c7d5ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:44.293210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.293175 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-dshm\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:44.293210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.293209 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r786f\" (UniqueName: \"kubernetes.io/projected/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-kube-api-access-r786f\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:44.293560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.293222 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tls-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:44.293560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.293237 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-home\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:44.293560 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.293247 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea-tmp-dir\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:44.539925 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.539886 4312 generic.go:356] "Generic (PLEG): container finished" podID="a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" containerID="ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a" exitCode=0 Nov 04 01:46:44.540119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.539944 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" Nov 04 01:46:44.540119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.539950 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" event={"ID":"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea","Type":"ContainerDied","Data":"ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a"} Nov 04 01:46:44.540119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.539989 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" event={"ID":"a72bce0f-3a71-44c7-ba8a-0db048c7d5ea","Type":"ContainerDied","Data":"cc8e5f42cc728eee96846237748e26ba53891c9a9c4f9fb236e7e2dc08f10bb4"} Nov 04 01:46:44.540119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.540018 4312 scope.go:122] "RemoveContainer" containerID="ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a" Nov 04 01:46:44.551260 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.551217 4312 scope.go:122] "RemoveContainer" containerID="ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a" Nov 04 01:46:44.552366 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:46:44.552312 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a\": container with ID starting with ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a not found: ID does not exist" containerID="ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a" Nov 04 01:46:44.552491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.552374 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a"} err="failed to get container status \"ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a\": rpc error: code = NotFound desc = could not find container \"ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a\": container with ID starting with ab658abbebbdd9354af334a6277fcb225752cf419d7fda9aa67446145097df2a not found: ID does not exist" Nov 04 01:46:44.569222 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.568743 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l"] Nov 04 01:46:44.572948 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:44.572921 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l"] Nov 04 01:46:45.046900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:45.046850 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-f3467868-kserve-fbf6d69b5-n765l" podUID="a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.222:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:46:46.150356 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:46.149685 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" path="/var/lib/kubelet/pods/a72bce0f-3a71-44c7-ba8a-0db048c7d5ea/volumes" Nov 04 01:46:47.516513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:47.516468 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:46:47.517161 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:47.517123 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-kvnxf" podUID="2ee19db5-0a09-4b15-b268-e749a0a2969f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:46:48.786827 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.786784 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh"] Nov 04 01:46:48.787498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787475 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" containerName="main" Nov 04 01:46:48.787613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787498 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" containerName="main" Nov 04 01:46:48.787613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787512 4312 cpu_manager.go:414] "containerMap: removing container" podUID="786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" containerName="revoke-keys" Nov 04 01:46:48.787613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787520 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" containerName="revoke-keys" Nov 04 01:46:48.787613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787553 4312 cpu_manager.go:414] "containerMap: removing container" podUID="103fab8e-782e-4c86-8c8f-d5eecbeb74d1" containerName="main" Nov 04 01:46:48.787613 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787560 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103fab8e-782e-4c86-8c8f-d5eecbeb74d1" containerName="main" Nov 04 01:46:48.787887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787665 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="103fab8e-782e-4c86-8c8f-d5eecbeb74d1" containerName="main" Nov 04 01:46:48.787887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787686 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72bce0f-3a71-44c7-ba8a-0db048c7d5ea" containerName="main" Nov 04 01:46:48.787887 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.787693 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" containerName="revoke-keys" Nov 04 01:46:48.791540 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.791516 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:48.796490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.795978 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:46:48.796490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.796003 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\"" Nov 04 01:46:48.796490 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.796056 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:46:48.796858 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.796836 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:46:48.797900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.797880 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh"] Nov 04 01:46:48.900279 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.900250 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:48.900460 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.900318 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5j2\" (UniqueName: \"kubernetes.io/projected/837ccc7d-89bb-4a35-bdfa-f015487b2664-kube-api-access-fn5j2\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:48.900519 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:48.900498 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.001407 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.001370 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.001587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.001444 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5j2\" (UniqueName: \"kubernetes.io/projected/837ccc7d-89bb-4a35-bdfa-f015487b2664-kube-api-access-fn5j2\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.001587 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.001505 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.002135 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.002110 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.003829 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.003805 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.014315 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.014292 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5j2\" (UniqueName: \"kubernetes.io/projected/837ccc7d-89bb-4a35-bdfa-f015487b2664-kube-api-access-fn5j2\") pod \"maas-api-revoke-sub-6e15d7d5-n9rrh\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.107553 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.107473 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:49.394276 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.394242 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh"] Nov 04 01:46:49.395397 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:46:49.395370 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837ccc7d_89bb_4a35_bdfa_f015487b2664.slice/crio-dc6a398090a1b4359261a1c585a1f7260f8e778f60779ba0542de61de09be4df WatchSource:0}: Error finding container dc6a398090a1b4359261a1c585a1f7260f8e778f60779ba0542de61de09be4df: Status 404 returned error can't find the container with id dc6a398090a1b4359261a1c585a1f7260f8e778f60779ba0542de61de09be4df Nov 04 01:46:49.613396 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:49.613358 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" event={"ID":"837ccc7d-89bb-4a35-bdfa-f015487b2664","Type":"ContainerStarted","Data":"dc6a398090a1b4359261a1c585a1f7260f8e778f60779ba0542de61de09be4df"} Nov 04 01:46:50.619698 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:50.619661 4312 generic.go:356] "Generic (PLEG): container finished" podID="837ccc7d-89bb-4a35-bdfa-f015487b2664" containerID="ded606ab9e04dea32a3f00b525e7de7728c828350b66ef7ce91735254f9b8282" exitCode=0 Nov 04 01:46:50.620104 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:50.619749 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" event={"ID":"837ccc7d-89bb-4a35-bdfa-f015487b2664","Type":"ContainerDied","Data":"ded606ab9e04dea32a3f00b525e7de7728c828350b66ef7ce91735254f9b8282"} Nov 04 01:46:51.751074 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.751048 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:51.828473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.828439 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-openshift-ca\") pod \"837ccc7d-89bb-4a35-bdfa-f015487b2664\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " Nov 04 01:46:51.828640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.828591 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-xks-ca\") pod \"837ccc7d-89bb-4a35-bdfa-f015487b2664\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " Nov 04 01:46:51.828689 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.828655 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837ccc7d-89bb-4a35-bdfa-f015487b2664-kube-api-access-fn5j2\" (UniqueName: \"kubernetes.io/projected/837ccc7d-89bb-4a35-bdfa-f015487b2664-kube-api-access-fn5j2\") pod \"837ccc7d-89bb-4a35-bdfa-f015487b2664\" (UID: \"837ccc7d-89bb-4a35-bdfa-f015487b2664\") " Nov 04 01:46:51.828785 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.828766 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-openshift-ca" pod "837ccc7d-89bb-4a35-bdfa-f015487b2664" (UID: "837ccc7d-89bb-4a35-bdfa-f015487b2664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:51.828972 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.828958 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-openshift-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:51.830767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.830737 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-xks-ca" pod "837ccc7d-89bb-4a35-bdfa-f015487b2664" (UID: "837ccc7d-89bb-4a35-bdfa-f015487b2664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:51.830767 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.830745 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837ccc7d-89bb-4a35-bdfa-f015487b2664-kube-api-access-fn5j2" pod "837ccc7d-89bb-4a35-bdfa-f015487b2664" (UID: "837ccc7d-89bb-4a35-bdfa-f015487b2664"). InnerVolumeSpecName "kube-api-access-fn5j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:51.930098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.930031 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5j2\" (UniqueName: \"kubernetes.io/projected/837ccc7d-89bb-4a35-bdfa-f015487b2664-kube-api-access-fn5j2\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:51.930098 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:51.930057 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837ccc7d-89bb-4a35-bdfa-f015487b2664-maas-api-xks-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:52.630987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:52.630954 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" event={"ID":"837ccc7d-89bb-4a35-bdfa-f015487b2664","Type":"ContainerDied","Data":"dc6a398090a1b4359261a1c585a1f7260f8e778f60779ba0542de61de09be4df"} Nov 04 01:46:52.631170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:52.630987 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh" Nov 04 01:46:52.631170 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:46:52.631002 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc6a398090a1b4359261a1c585a1f7260f8e778f60779ba0542de61de09be4df" Nov 04 01:46:59.460606 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:46:59.460521 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:47:01.024677 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.024643 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-668dc6579-bfxrx"] Nov 04 01:47:01.025617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.025585 4312 cpu_manager.go:414] "containerMap: removing container" podUID="837ccc7d-89bb-4a35-bdfa-f015487b2664" containerName="revoke-keys" Nov 04 01:47:01.025617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.025612 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837ccc7d-89bb-4a35-bdfa-f015487b2664" containerName="revoke-keys" Nov 04 01:47:01.025799 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.025753 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="837ccc7d-89bb-4a35-bdfa-f015487b2664" containerName="revoke-keys" Nov 04 01:47:01.029141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.029115 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.038879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.038783 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:47:01.041697 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.040054 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:47:01.044058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.044032 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-668dc6579-bfxrx"] Nov 04 01:47:01.044289 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.044257 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 01:47:01.045516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.045494 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rbkg2\"" Nov 04 01:47:01.045634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.045534 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:47:01.048311 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.048291 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx"] Nov 04 01:47:01.053854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.053831 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.065080 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.065054 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx"] Nov 04 01:47:01.123269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123234 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-maas-api-tls\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.123452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123289 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6f126-a7d2-4675-a9fd-980418dd31f1-openshift-service-ca\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.123452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123404 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feca7d9-7167-4353-b078-cf7f9278df85-openshift-service-ca\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.123452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123435 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-metrics-certs\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.123581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123545 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4feca7d9-7167-4353-b078-cf7f9278df85-maas-api-tls\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.123640 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123614 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7xh\" (UniqueName: \"kubernetes.io/projected/1ee6f126-a7d2-4675-a9fd-980418dd31f1-kube-api-access-xs7xh\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.123687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123670 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchqw\" (UniqueName: \"kubernetes.io/projected/4feca7d9-7167-4353-b078-cf7f9278df85-kube-api-access-tchqw\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.123734 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.123712 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4feca7d9-7167-4353-b078-cf7f9278df85-metrics-certs\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.225087 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225052 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4feca7d9-7167-4353-b078-cf7f9278df85-maas-api-tls\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.225268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225112 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7xh\" (UniqueName: \"kubernetes.io/projected/1ee6f126-a7d2-4675-a9fd-980418dd31f1-kube-api-access-xs7xh\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.225268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225130 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchqw\" (UniqueName: \"kubernetes.io/projected/4feca7d9-7167-4353-b078-cf7f9278df85-kube-api-access-tchqw\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.225268 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225245 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4feca7d9-7167-4353-b078-cf7f9278df85-metrics-certs\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.225522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225362 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-maas-api-tls\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.225522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225395 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6f126-a7d2-4675-a9fd-980418dd31f1-openshift-service-ca\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.225522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225422 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feca7d9-7167-4353-b078-cf7f9278df85-openshift-service-ca\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.225522 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.225451 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-metrics-certs\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.226114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.226086 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6f126-a7d2-4675-a9fd-980418dd31f1-openshift-service-ca\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.226235 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.226133 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4feca7d9-7167-4353-b078-cf7f9278df85-openshift-service-ca\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.227815 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.227790 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4feca7d9-7167-4353-b078-cf7f9278df85-metrics-certs\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.227924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.227798 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-metrics-certs\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.228037 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.228021 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4feca7d9-7167-4353-b078-cf7f9278df85-maas-api-tls\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.228314 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.228296 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-maas-api-tls\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.237632 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.237608 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7xh\" (UniqueName: \"kubernetes.io/projected/1ee6f126-a7d2-4675-a9fd-980418dd31f1-kube-api-access-xs7xh\") pod \"maas-api-668dc6579-bfxrx\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.237722 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.237631 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchqw\" (UniqueName: \"kubernetes.io/projected/4feca7d9-7167-4353-b078-cf7f9278df85-kube-api-access-tchqw\") pod \"maas-api-bd788d5f-cwfvx\" (UID: \"4feca7d9-7167-4353-b078-cf7f9278df85\") " pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.347826 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.347791 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:01.381696 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.381660 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:01.635019 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.634989 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-668dc6579-bfxrx"] Nov 04 01:47:01.636378 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:01.636348 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee6f126_a7d2_4675_a9fd_980418dd31f1.slice/crio-7f4efa55b9d17070ce522bdf3a1720a0fa28baf7249fe210ee2a8916dd26a439 WatchSource:0}: Error finding container 7f4efa55b9d17070ce522bdf3a1720a0fa28baf7249fe210ee2a8916dd26a439: Status 404 returned error can't find the container with id 7f4efa55b9d17070ce522bdf3a1720a0fa28baf7249fe210ee2a8916dd26a439 Nov 04 01:47:01.637655 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.637636 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:47:01.678391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.678357 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" event={"ID":"1ee6f126-a7d2-4675-a9fd-980418dd31f1","Type":"ContainerStarted","Data":"7f4efa55b9d17070ce522bdf3a1720a0fa28baf7249fe210ee2a8916dd26a439"} Nov 04 01:47:01.682320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:01.682294 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx"] Nov 04 01:47:02.685469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:02.685427 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" event={"ID":"1ee6f126-a7d2-4675-a9fd-980418dd31f1","Type":"ContainerStarted","Data":"0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499"} Nov 04 01:47:02.685906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:02.685555 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:02.687516 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:02.687490 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" event={"ID":"4feca7d9-7167-4353-b078-cf7f9278df85","Type":"ContainerStarted","Data":"d0353f3056cc63eff50ba0c2662379b200993db4985b2cfec413bf18bee77d76"} Nov 04 01:47:02.687672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:02.687532 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:02.687672 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:02.687548 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" event={"ID":"4feca7d9-7167-4353-b078-cf7f9278df85","Type":"ContainerStarted","Data":"d15b7659f4df22c4c1aeb1d91070d5ad0310f1bf8a3594c8f548e075015b18f0"} Nov 04 01:47:02.707180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:02.707136 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" podStartSLOduration=1.247910177 podStartE2EDuration="1.707125114s" totalImagesPullingTime="459.214937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:02.705437466 +0000 UTC m=+2654.728369493" watchObservedRunningTime="2026-11-04 01:47:02.707125114 +0000 UTC m=+2654.730057120" Nov 04 01:47:02.727359 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:02.727299 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" podStartSLOduration=1.196253689 podStartE2EDuration="1.727285371s" totalImagesPullingTime="531.031682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:02.725887803 +0000 UTC m=+2654.748819809" watchObservedRunningTime="2026-11-04 01:47:02.727285371 +0000 UTC m=+2654.750217377" Nov 04 01:47:08.208834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.208787 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8"] Nov 04 01:47:08.213764 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.213739 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.221128 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.221050 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-39635e51-openshift-default-dockercfg-rh5td\"" Nov 04 01:47:08.225124 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.225101 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8"] Nov 04 01:47:08.295637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.295597 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-podinfo\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.295832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.295655 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-token\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.295832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.295760 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istiod-ca-cert\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.295832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.295812 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-data\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.295982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.295868 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-certs\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.295982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.295925 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-socket\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.295982 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.295948 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vbb\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-kube-api-access-r8vbb\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.296105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.296027 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-envoy\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.296105 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.296051 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-credential-socket\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.396792 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.396752 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-envoy\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.396979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.396804 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-credential-socket\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.396979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.396842 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-podinfo\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.396979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.396892 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-token\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.396979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.396935 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istiod-ca-cert\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.396979 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.396968 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-data\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.396995 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-certs\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.397023 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-socket\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.397061 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vbb\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-kube-api-access-r8vbb\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.397444 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-certs\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397507 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.397467 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-socket\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397546 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.397530 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-data\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397595 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.397571 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-credential-socket\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.397895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.397864 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istiod-ca-cert\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.399458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.399429 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-envoy\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.399596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.399576 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-podinfo\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.409400 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.409375 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-token\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.409498 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.409446 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vbb\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-kube-api-access-r8vbb\") pod \"e2e-ait-39635e51-openshift-default-98b9df875-6kqz8\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.529537 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.529514 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:08.697994 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.697928 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:08.698477 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.698456 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bd788d5f-cwfvx" Nov 04 01:47:08.777277 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.777235 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-668dc6579-bfxrx"] Nov 04 01:47:08.777545 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.777514 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" podUID="1ee6f126-a7d2-4675-a9fd-980418dd31f1" containerName="maas-api" containerID="cri-o://0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499" gracePeriod=30 Nov 04 01:47:08.816488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.816457 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8"] Nov 04 01:47:08.839553 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:08.839511 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cf86b4_99fa_46d1_98e5_4ffc4ccc2d84.slice/crio-7b3f7d340b19a8ee0fb6b2f835c431c95bca42d434fe69cc97cb4733e587a656 WatchSource:0}: Error finding container 7b3f7d340b19a8ee0fb6b2f835c431c95bca42d434fe69cc97cb4733e587a656: Status 404 returned error can't find the container with id 7b3f7d340b19a8ee0fb6b2f835c431c95bca42d434fe69cc97cb4733e587a656 Nov 04 01:47:08.885267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.884356 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:47:08.885267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.884454 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:47:08.885267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:08.884493 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:47:09.019929 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.019904 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:09.104713 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.104637 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-maas-api-tls\") pod \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " Nov 04 01:47:09.104865 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.104712 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee6f126-a7d2-4675-a9fd-980418dd31f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6f126-a7d2-4675-a9fd-980418dd31f1-openshift-service-ca\") pod \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " Nov 04 01:47:09.104936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.104888 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee6f126-a7d2-4675-a9fd-980418dd31f1-kube-api-access-xs7xh\" (UniqueName: \"kubernetes.io/projected/1ee6f126-a7d2-4675-a9fd-980418dd31f1-kube-api-access-xs7xh\") pod \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " Nov 04 01:47:09.104936 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.104928 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-metrics-certs\") pod \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\" (UID: \"1ee6f126-a7d2-4675-a9fd-980418dd31f1\") " Nov 04 01:47:09.105344 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.105302 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee6f126-a7d2-4675-a9fd-980418dd31f1-openshift-service-ca" pod "1ee6f126-a7d2-4675-a9fd-980418dd31f1" (UID: "1ee6f126-a7d2-4675-a9fd-980418dd31f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:09.107828 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.107797 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-maas-api-tls" pod "1ee6f126-a7d2-4675-a9fd-980418dd31f1" (UID: "1ee6f126-a7d2-4675-a9fd-980418dd31f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:09.107930 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.107912 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee6f126-a7d2-4675-a9fd-980418dd31f1-kube-api-access-xs7xh" pod "1ee6f126-a7d2-4675-a9fd-980418dd31f1" (UID: "1ee6f126-a7d2-4675-a9fd-980418dd31f1"). InnerVolumeSpecName "kube-api-access-xs7xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:09.108448 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.108417 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-metrics-certs" pod "1ee6f126-a7d2-4675-a9fd-980418dd31f1" (UID: "1ee6f126-a7d2-4675-a9fd-980418dd31f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:09.206392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.206355 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6f126-a7d2-4675-a9fd-980418dd31f1-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:09.206392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.206383 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7xh\" (UniqueName: \"kubernetes.io/projected/1ee6f126-a7d2-4675-a9fd-980418dd31f1-kube-api-access-xs7xh\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:09.206392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.206393 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:09.206392 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.206402 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee6f126-a7d2-4675-a9fd-980418dd31f1-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:09.728597 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.728505 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" event={"ID":"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84","Type":"ContainerStarted","Data":"50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac"} Nov 04 01:47:09.728597 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.728569 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" event={"ID":"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84","Type":"ContainerStarted","Data":"7b3f7d340b19a8ee0fb6b2f835c431c95bca42d434fe69cc97cb4733e587a656"} Nov 04 01:47:09.729893 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.729868 4312 generic.go:356] "Generic (PLEG): container finished" podID="1ee6f126-a7d2-4675-a9fd-980418dd31f1" containerID="0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499" exitCode=0 Nov 04 01:47:09.730013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.729904 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" event={"ID":"1ee6f126-a7d2-4675-a9fd-980418dd31f1","Type":"ContainerDied","Data":"0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499"} Nov 04 01:47:09.730013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.729916 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" Nov 04 01:47:09.730013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.729933 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-668dc6579-bfxrx" event={"ID":"1ee6f126-a7d2-4675-a9fd-980418dd31f1","Type":"ContainerDied","Data":"7f4efa55b9d17070ce522bdf3a1720a0fa28baf7249fe210ee2a8916dd26a439"} Nov 04 01:47:09.730013 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.729952 4312 scope.go:122] "RemoveContainer" containerID="0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499" Nov 04 01:47:09.739569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.739551 4312 scope.go:122] "RemoveContainer" containerID="0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499" Nov 04 01:47:09.739829 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:09.739803 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499\": container with ID starting with 0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499 not found: ID does not exist" containerID="0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499" Nov 04 01:47:09.739889 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.739827 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499"} err="failed to get container status \"0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499\": rpc error: code = NotFound desc = could not find container \"0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499\": container with ID starting with 0361d117b0049987d48594477bf687881b41e7294ef7d65b2d2430b98ef78499 not found: ID does not exist" Nov 04 01:47:09.753382 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.753318 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podStartSLOduration=1.753285747 podStartE2EDuration="1.753285747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:09.753133168 +0000 UTC m=+2661.776065175" watchObservedRunningTime="2026-11-04 01:47:09.753285747 +0000 UTC m=+2661.776217753" Nov 04 01:47:09.768202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.768176 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-668dc6579-bfxrx"] Nov 04 01:47:09.772015 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:09.771994 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-668dc6579-bfxrx"] Nov 04 01:47:10.108878 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:10.108844 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee6f126-a7d2-4675-a9fd-980418dd31f1" path="/var/lib/kubelet/pods/1ee6f126-a7d2-4675-a9fd-980418dd31f1/volumes" Nov 04 01:47:10.530160 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:10.530126 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:10.535187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:10.535164 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:10.743592 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:10.743555 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:10.744585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:10.744559 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:12.535928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.535895 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86"] Nov 04 01:47:12.536367 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.536352 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1ee6f126-a7d2-4675-a9fd-980418dd31f1" containerName="maas-api" Nov 04 01:47:12.536421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.536367 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee6f126-a7d2-4675-a9fd-980418dd31f1" containerName="maas-api" Nov 04 01:47:12.536455 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.536442 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee6f126-a7d2-4675-a9fd-980418dd31f1" containerName="maas-api" Nov 04 01:47:12.541000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.540981 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.547726 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.547694 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-39635e51-serving-cert\"" Nov 04 01:47:12.552185 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.552161 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86"] Nov 04 01:47:12.642390 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.642360 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-metrics-certs\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.642581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.642435 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3775cccb-abcd-441b-8911-828ebe37f451-openshift-service-ca\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.642581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.642566 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6dp\" (UniqueName: \"kubernetes.io/projected/3775cccb-abcd-441b-8911-828ebe37f451-kube-api-access-2z6dp\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.642687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.642639 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-maas-api-tls\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.678274 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.678244 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w"] Nov 04 01:47:12.681984 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.681962 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:12.691755 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.691728 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w"] Nov 04 01:47:12.692675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.692242 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-39635e51\"" Nov 04 01:47:12.717475 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.717446 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j"] Nov 04 01:47:12.721820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.721804 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:12.729258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.729223 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-39635e51-dockercfg-jtjc7\"" Nov 04 01:47:12.731287 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.731263 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j"] Nov 04 01:47:12.743751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.743730 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-metrics-certs\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.743863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.743777 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3775cccb-abcd-441b-8911-828ebe37f451-openshift-service-ca\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.743863 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.743839 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6dp\" (UniqueName: \"kubernetes.io/projected/3775cccb-abcd-441b-8911-828ebe37f451-kube-api-access-2z6dp\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.743966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.743863 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-maas-api-tls\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.744645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.744619 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3775cccb-abcd-441b-8911-828ebe37f451-openshift-service-ca\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.746427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.746407 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-metrics-certs\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.746619 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.746600 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-maas-api-tls\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.757898 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.757873 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6dp\" (UniqueName: \"kubernetes.io/projected/3775cccb-abcd-441b-8911-828ebe37f451-kube-api-access-2z6dp\") pod \"maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.845598 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.845486 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2gt\" (UniqueName: \"kubernetes.io/projected/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-kube-api-access-tc2gt\") pod \"payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:12.845789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.845654 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjrd\" (UniqueName: \"kubernetes.io/projected/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-kube-api-access-vhjrd\") pod \"payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:12.845789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.845679 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:12.845789 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.845697 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:12.858665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.858640 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:12.947016 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.946980 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2gt\" (UniqueName: \"kubernetes.io/projected/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-kube-api-access-tc2gt\") pod \"payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:12.947177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.947101 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjrd\" (UniqueName: \"kubernetes.io/projected/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-kube-api-access-vhjrd\") pod \"payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:12.947177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.947134 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:12.947177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.947160 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:12.947871 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.947848 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:12.947969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.947869 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:12.959594 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.959568 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjrd\" (UniqueName: \"kubernetes.io/projected/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-kube-api-access-vhjrd\") pod \"payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:12.959710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.959689 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2gt\" (UniqueName: \"kubernetes.io/projected/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-kube-api-access-tc2gt\") pod \"payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:12.997561 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:12.997539 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:13.035429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.035400 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:13.140964 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.140758 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86"] Nov 04 01:47:13.142871 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:13.142843 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3775cccb_abcd_441b_8911_828ebe37f451.slice/crio-146ba14f6b3ff781a121739a1997274acdc93312885b81d4cd41178a4a7340f3 WatchSource:0}: Error finding container 146ba14f6b3ff781a121739a1997274acdc93312885b81d4cd41178a4a7340f3: Status 404 returned error can't find the container with id 146ba14f6b3ff781a121739a1997274acdc93312885b81d4cd41178a4a7340f3 Nov 04 01:47:13.310739 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.310711 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w"] Nov 04 01:47:13.312049 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:13.312020 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a5cf1f_7d5e_465a_ba32_08a89016a1c3.slice/crio-73efdc389461bc7acf1f562c30eb508ccc0054c3d19ee31bdec095347ab46d3c WatchSource:0}: Error finding container 73efdc389461bc7acf1f562c30eb508ccc0054c3d19ee31bdec095347ab46d3c: Status 404 returned error can't find the container with id 73efdc389461bc7acf1f562c30eb508ccc0054c3d19ee31bdec095347ab46d3c Nov 04 01:47:13.350527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.350500 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j"] Nov 04 01:47:13.352252 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:13.352209 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdced3f_a08b_41fb_b59e_0eaf32fe68f1.slice/crio-05ab8a12663cd9b2ce61ab5bb883f70e3c29ba5d4307a9975dcf3df501e74fd9 WatchSource:0}: Error finding container 05ab8a12663cd9b2ce61ab5bb883f70e3c29ba5d4307a9975dcf3df501e74fd9: Status 404 returned error can't find the container with id 05ab8a12663cd9b2ce61ab5bb883f70e3c29ba5d4307a9975dcf3df501e74fd9 Nov 04 01:47:13.764600 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.764559 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" event={"ID":"3775cccb-abcd-441b-8911-828ebe37f451","Type":"ContainerStarted","Data":"ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5"} Nov 04 01:47:13.765180 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.764662 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" event={"ID":"3775cccb-abcd-441b-8911-828ebe37f451","Type":"ContainerStarted","Data":"146ba14f6b3ff781a121739a1997274acdc93312885b81d4cd41178a4a7340f3"} Nov 04 01:47:13.765464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.765439 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:13.767410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.767383 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" event={"ID":"afdced3f-a08b-41fb-b59e-0eaf32fe68f1","Type":"ContainerStarted","Data":"4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244"} Nov 04 01:47:13.767543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.767436 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" event={"ID":"afdced3f-a08b-41fb-b59e-0eaf32fe68f1","Type":"ContainerStarted","Data":"05ab8a12663cd9b2ce61ab5bb883f70e3c29ba5d4307a9975dcf3df501e74fd9"} Nov 04 01:47:13.767543 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.767481 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:13.769248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.769222 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" event={"ID":"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3","Type":"ContainerStarted","Data":"e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2"} Nov 04 01:47:13.769372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.769263 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" event={"ID":"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3","Type":"ContainerStarted","Data":"73efdc389461bc7acf1f562c30eb508ccc0054c3d19ee31bdec095347ab46d3c"} Nov 04 01:47:13.769372 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.769363 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:13.788183 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.788129 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" podStartSLOduration=1.287709363 podStartE2EDuration="1.788112604s" totalImagesPullingTime="500.403241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:13.784655645 +0000 UTC m=+2665.807587654" watchObservedRunningTime="2026-11-04 01:47:13.788112604 +0000 UTC m=+2665.811044610" Nov 04 01:47:13.806756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.806719 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" podStartSLOduration=1.8067077070000002 podStartE2EDuration="1.806707707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:13.806099385 +0000 UTC m=+2665.829031411" watchObservedRunningTime="2026-11-04 01:47:13.806707707 +0000 UTC m=+2665.829639713" Nov 04 01:47:13.831413 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:13.831374 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" podStartSLOduration=1.8313630920000001 podStartE2EDuration="1.831363092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:13.8287353 +0000 UTC m=+2665.851667309" watchObservedRunningTime="2026-11-04 01:47:13.831363092 +0000 UTC m=+2665.854295099" Nov 04 01:47:20.787602 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:20.787573 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:24.779267 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:24.779241 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:24.780521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:24.779306 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:25.745782 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.745747 4312 patch_prober.go:29] interesting pod/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Nov 04 01:47:25.745980 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.745814 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Nov 04 01:47:25.839422 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.839388 4312 generic.go:356] "Generic (PLEG): container finished" podID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerID="50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac" exitCode=0 Nov 04 01:47:25.839802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.839457 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" event={"ID":"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84","Type":"ContainerDied","Data":"50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac"} Nov 04 01:47:25.839900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.839885 4312 scope.go:122] "RemoveContainer" containerID="50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac" Nov 04 01:47:25.841115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.841088 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:47:25.841187 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.841158 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:47:25.841225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:25.841188 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 04 01:47:26.846968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:26.846930 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" event={"ID":"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84","Type":"ContainerStarted","Data":"3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2"} Nov 04 01:47:27.530430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:27.530391 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:27.530773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:27.530616 4312 patch_prober.go:29] interesting pod/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Nov 04 01:47:27.530773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:27.530673 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Nov 04 01:47:28.530624 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:28.530588 4312 patch_prober.go:29] interesting pod/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Nov 04 01:47:28.531123 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:28.530652 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Nov 04 01:47:29.477809 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:29.477770 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:47:29.530671 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:29.530640 4312 patch_prober.go:29] interesting pod/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Nov 04 01:47:29.531050 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:29.530720 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Nov 04 01:47:30.530645 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:30.530612 4312 patch_prober.go:29] interesting pod/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Nov 04 01:47:30.530826 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:30.530687 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Nov 04 01:47:31.530171 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:31.530133 4312 patch_prober.go:29] interesting pod/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Nov 04 01:47:31.530429 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:31.530195 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Nov 04 01:47:32.534700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:32.534667 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:32.535100 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:32.535081 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:32.535859 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:32.535840 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:37.293085 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.293045 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86"] Nov 04 01:47:37.293654 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.293260 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" podUID="3775cccb-abcd-441b-8911-828ebe37f451" containerName="maas-api" containerID="cri-o://ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5" gracePeriod=30 Nov 04 01:47:37.314551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.314522 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w"] Nov 04 01:47:37.334895 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.334864 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j"] Nov 04 01:47:37.603854 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.603822 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:37.715463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.715426 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-metrics-certs\") pod \"3775cccb-abcd-441b-8911-828ebe37f451\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " Nov 04 01:47:37.715688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.715499 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3775cccb-abcd-441b-8911-828ebe37f451-kube-api-access-2z6dp\" (UniqueName: \"kubernetes.io/projected/3775cccb-abcd-441b-8911-828ebe37f451-kube-api-access-2z6dp\") pod \"3775cccb-abcd-441b-8911-828ebe37f451\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " Nov 04 01:47:37.715688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.715655 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-maas-api-tls\") pod \"3775cccb-abcd-441b-8911-828ebe37f451\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " Nov 04 01:47:37.715821 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.715699 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3775cccb-abcd-441b-8911-828ebe37f451-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3775cccb-abcd-441b-8911-828ebe37f451-openshift-service-ca\") pod \"3775cccb-abcd-441b-8911-828ebe37f451\" (UID: \"3775cccb-abcd-441b-8911-828ebe37f451\") " Nov 04 01:47:37.716463 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.716429 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3775cccb-abcd-441b-8911-828ebe37f451-openshift-service-ca" pod "3775cccb-abcd-441b-8911-828ebe37f451" (UID: "3775cccb-abcd-441b-8911-828ebe37f451"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:37.718616 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.718582 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3775cccb-abcd-441b-8911-828ebe37f451-kube-api-access-2z6dp" pod "3775cccb-abcd-441b-8911-828ebe37f451" (UID: "3775cccb-abcd-441b-8911-828ebe37f451"). InnerVolumeSpecName "kube-api-access-2z6dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:37.718744 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.718640 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-maas-api-tls" pod "3775cccb-abcd-441b-8911-828ebe37f451" (UID: "3775cccb-abcd-441b-8911-828ebe37f451"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:37.719065 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.719038 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-metrics-certs" pod "3775cccb-abcd-441b-8911-828ebe37f451" (UID: "3775cccb-abcd-441b-8911-828ebe37f451"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:37.816981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.816899 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:37.816981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.816929 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3775cccb-abcd-441b-8911-828ebe37f451-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:37.816981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.816938 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3775cccb-abcd-441b-8911-828ebe37f451-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:37.816981 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.816948 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6dp\" (UniqueName: \"kubernetes.io/projected/3775cccb-abcd-441b-8911-828ebe37f451-kube-api-access-2z6dp\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:37.926762 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.926731 4312 generic.go:356] "Generic (PLEG): container finished" podID="3775cccb-abcd-441b-8911-828ebe37f451" containerID="ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5" exitCode=0 Nov 04 01:47:37.926931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.926786 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" event={"ID":"3775cccb-abcd-441b-8911-828ebe37f451","Type":"ContainerDied","Data":"ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5"} Nov 04 01:47:37.926931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.926828 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" event={"ID":"3775cccb-abcd-441b-8911-828ebe37f451","Type":"ContainerDied","Data":"146ba14f6b3ff781a121739a1997274acdc93312885b81d4cd41178a4a7340f3"} Nov 04 01:47:37.926931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.926849 4312 scope.go:122] "RemoveContainer" containerID="ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5" Nov 04 01:47:37.926931 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.926787 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86" Nov 04 01:47:37.937304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.937281 4312 scope.go:122] "RemoveContainer" containerID="ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5" Nov 04 01:47:37.937603 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:37.937579 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5\": container with ID starting with ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5 not found: ID does not exist" containerID="ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5" Nov 04 01:47:37.937693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.937610 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5"} err="failed to get container status \"ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5\": rpc error: code = NotFound desc = could not find container \"ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5\": container with ID starting with ea6b77df22476ad2e2b999065836eb4ef96008a51b37c26440b61e6d9913c5e5 not found: ID does not exist" Nov 04 01:47:37.951716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.951694 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86"] Nov 04 01:47:37.954209 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:37.954184 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39635e51-6bcc747ccb-j6t86"] Nov 04 01:47:38.110114 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:38.110037 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3775cccb-abcd-441b-8911-828ebe37f451" path="/var/lib/kubelet/pods/3775cccb-abcd-441b-8911-828ebe37f451/volumes" Nov 04 01:47:38.459941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:38.459845 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8"] Nov 04 01:47:38.460499 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:38.460106 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" containerID="cri-o://3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2" gracePeriod=30 Nov 04 01:47:42.315500 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.315450 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" podUID="e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" containerName="payload-processing" containerID="cri-o://e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2" gracePeriod=25 Nov 04 01:47:42.336075 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.336022 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" podUID="afdced3f-a08b-41fb-b59e-0eaf32fe68f1" containerName="payload-pre-processing" containerID="cri-o://4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244" gracePeriod=25 Nov 04 01:47:42.598028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.597672 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:42.630493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.630465 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:42.669120 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.669087 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-kube-api-access-tc2gt\" (UniqueName: \"kubernetes.io/projected/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-kube-api-access-tc2gt\") pod \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " Nov 04 01:47:42.669318 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.669128 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-plugins-config-volume\") pod \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\" (UID: \"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3\") " Nov 04 01:47:42.669633 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.669605 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-plugins-config-volume" pod "e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" (UID: "e2a5cf1f-7d5e-465a-ba32-08a89016a1c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:42.671141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.671116 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-kube-api-access-tc2gt" pod "e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" (UID: "e2a5cf1f-7d5e-465a-ba32-08a89016a1c3"). InnerVolumeSpecName "kube-api-access-tc2gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:42.770693 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.770664 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-plugins-config-volume\") pod \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " Nov 04 01:47:42.770904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.770749 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-kube-api-access-vhjrd\" (UniqueName: \"kubernetes.io/projected/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-kube-api-access-vhjrd\") pod \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\" (UID: \"afdced3f-a08b-41fb-b59e-0eaf32fe68f1\") " Nov 04 01:47:42.771039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.771023 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2gt\" (UniqueName: \"kubernetes.io/projected/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-kube-api-access-tc2gt\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:42.771099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.771041 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:42.771099 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.771077 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-plugins-config-volume" pod "afdced3f-a08b-41fb-b59e-0eaf32fe68f1" (UID: "afdced3f-a08b-41fb-b59e-0eaf32fe68f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:42.772928 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.772898 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-kube-api-access-vhjrd" pod "afdced3f-a08b-41fb-b59e-0eaf32fe68f1" (UID: "afdced3f-a08b-41fb-b59e-0eaf32fe68f1"). InnerVolumeSpecName "kube-api-access-vhjrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:42.871746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.871651 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:42.871746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.871683 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjrd\" (UniqueName: \"kubernetes.io/projected/afdced3f-a08b-41fb-b59e-0eaf32fe68f1-kube-api-access-vhjrd\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:42.956164 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.956126 4312 generic.go:356] "Generic (PLEG): container finished" podID="e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" containerID="e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2" exitCode=0 Nov 04 01:47:42.956415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.956178 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" Nov 04 01:47:42.956415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.956221 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" event={"ID":"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3","Type":"ContainerDied","Data":"e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2"} Nov 04 01:47:42.956415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.956277 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w" event={"ID":"e2a5cf1f-7d5e-465a-ba32-08a89016a1c3","Type":"ContainerDied","Data":"73efdc389461bc7acf1f562c30eb508ccc0054c3d19ee31bdec095347ab46d3c"} Nov 04 01:47:42.956415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.956311 4312 scope.go:122] "RemoveContainer" containerID="e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2" Nov 04 01:47:42.957961 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.957934 4312 generic.go:356] "Generic (PLEG): container finished" podID="afdced3f-a08b-41fb-b59e-0eaf32fe68f1" containerID="4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244" exitCode=0 Nov 04 01:47:42.958096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.957999 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" Nov 04 01:47:42.958096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.958020 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" event={"ID":"afdced3f-a08b-41fb-b59e-0eaf32fe68f1","Type":"ContainerDied","Data":"4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244"} Nov 04 01:47:42.958096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.958082 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j" event={"ID":"afdced3f-a08b-41fb-b59e-0eaf32fe68f1","Type":"ContainerDied","Data":"05ab8a12663cd9b2ce61ab5bb883f70e3c29ba5d4307a9975dcf3df501e74fd9"} Nov 04 01:47:42.968869 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.968850 4312 scope.go:122] "RemoveContainer" containerID="e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2" Nov 04 01:47:42.969236 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:42.969179 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2\": container with ID starting with e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2 not found: ID does not exist" containerID="e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2" Nov 04 01:47:42.969376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.969271 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2"} err="failed to get container status \"e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2\": rpc error: code = NotFound desc = could not find container \"e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2\": container with ID starting with e623c0e01060d5933c133480a9b09f5fbe0f70a6c80d57ccc78829e36bbffaa2 not found: ID does not exist" Nov 04 01:47:42.969376 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.969297 4312 scope.go:122] "RemoveContainer" containerID="4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244" Nov 04 01:47:42.983415 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.983390 4312 scope.go:122] "RemoveContainer" containerID="4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244" Nov 04 01:47:42.983737 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:42.983719 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244\": container with ID starting with 4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244 not found: ID does not exist" containerID="4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244" Nov 04 01:47:42.983832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.983745 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244"} err="failed to get container status \"4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244\": rpc error: code = NotFound desc = could not find container \"4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244\": container with ID starting with 4088dae53d1f703289888cebdf5f44c6fec079098482e653c546df7979f35244 not found: ID does not exist" Nov 04 01:47:42.984151 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.984129 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w"] Nov 04 01:47:42.988153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.988131 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39635e51-76bc945fc6-smc6w"] Nov 04 01:47:42.999890 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:42.999870 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j"] Nov 04 01:47:43.006864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.006842 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39635e51-6bd676f947-pzh9j"] Nov 04 01:47:43.724039 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.724013 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:43.883113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883033 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-certs\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883092 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-credential-socket\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883113 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883114 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-socket\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883147 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-podinfo\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883190 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istiod-ca-cert\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883210 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-envoy\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883227 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-data\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883309 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-token\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-token\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883357 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-certs" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883401 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-kube-api-access-r8vbb\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-kube-api-access-r8vbb\") pod \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\" (UID: \"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84\") " Nov 04 01:47:43.883464 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883422 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-credential-socket" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:43.883937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883513 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-socket" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:43.883937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883672 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istiod-ca-cert" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:43.883937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883796 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-credential-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.883937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883816 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-socket\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.883937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883830 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istiod-ca-cert\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.883937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883843 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-workload-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.883937 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.883908 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-data" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:43.885552 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.885524 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-envoy" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:43.885657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.885554 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-podinfo" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:47:43.885657 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.885594 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-token" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:43.885736 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.885664 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-kube-api-access-r8vbb" pod "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" (UID: "24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84"). InnerVolumeSpecName "kube-api-access-r8vbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:43.964750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.964717 4312 generic.go:356] "Generic (PLEG): container finished" podID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerID="3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2" exitCode=0 Nov 04 01:47:43.964880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.964757 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" Nov 04 01:47:43.964880 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.964828 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" event={"ID":"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84","Type":"ContainerDied","Data":"3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2"} Nov 04 01:47:43.965008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.964878 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8" event={"ID":"24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84","Type":"ContainerDied","Data":"7b3f7d340b19a8ee0fb6b2f835c431c95bca42d434fe69cc97cb4733e587a656"} Nov 04 01:47:43.965008 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.964911 4312 scope.go:122] "RemoveContainer" containerID="3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2" Nov 04 01:47:43.982716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.982696 4312 scope.go:122] "RemoveContainer" containerID="50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac" Nov 04 01:47:43.984272 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.984249 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-envoy\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.984272 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.984270 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-data\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.984472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.984279 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-token\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.984472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.984289 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vbb\" (UniqueName: \"kubernetes.io/projected/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-kube-api-access-r8vbb\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.984472 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.984298 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84-istio-podinfo\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:43.991258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.991230 4312 scope.go:122] "RemoveContainer" containerID="3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2" Nov 04 01:47:43.991565 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:43.991533 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2\": container with ID starting with 3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2 not found: ID does not exist" containerID="3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2" Nov 04 01:47:43.991651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.991573 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2"} err="failed to get container status \"3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2\": rpc error: code = NotFound desc = could not find container \"3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2\": container with ID starting with 3036fdf5c074e89cc1f8daa26316023239eafc3cc76223c819e2aa136d4c88c2 not found: ID does not exist" Nov 04 01:47:43.991651 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.991596 4312 scope.go:122] "RemoveContainer" containerID="50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac" Nov 04 01:47:43.991865 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:43.991832 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac\": container with ID starting with 50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac not found: ID does not exist" containerID="50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac" Nov 04 01:47:43.991924 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:43.991870 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac"} err="failed to get container status \"50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac\": rpc error: code = NotFound desc = could not find container \"50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac\": container with ID starting with 50666709dbf17e138624919f0c72a4d3e90cc5b0ccdef871b0f9b515371feaac not found: ID does not exist" Nov 04 01:47:44.000488 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.000467 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8"] Nov 04 01:47:44.004058 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.004030 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-39635e51-openshift-default-98b9df875-6kqz8"] Nov 04 01:47:44.015277 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:44.015255 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cf86b4_99fa_46d1_98e5_4ffc4ccc2d84.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:47:44.107641 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.107606 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" path="/var/lib/kubelet/pods/24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84/volumes" Nov 04 01:47:44.107997 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.107982 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdced3f-a08b-41fb-b59e-0eaf32fe68f1" path="/var/lib/kubelet/pods/afdced3f-a08b-41fb-b59e-0eaf32fe68f1/volumes" Nov 04 01:47:44.108345 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.108322 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" path="/var/lib/kubelet/pods/e2a5cf1f-7d5e-465a-ba32-08a89016a1c3/volumes" Nov 04 01:47:44.824807 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.824765 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc"] Nov 04 01:47:44.825783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825754 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" containerName="payload-processing" Nov 04 01:47:44.825783 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825782 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" containerName="payload-processing" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825820 4312 cpu_manager.go:414] "containerMap: removing container" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825828 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825849 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3775cccb-abcd-441b-8911-828ebe37f451" containerName="maas-api" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825856 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3775cccb-abcd-441b-8911-828ebe37f451" containerName="maas-api" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825899 4312 cpu_manager.go:414] "containerMap: removing container" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825906 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825914 4312 cpu_manager.go:414] "containerMap: removing container" podUID="afdced3f-a08b-41fb-b59e-0eaf32fe68f1" containerName="payload-pre-processing" Nov 04 01:47:44.825962 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.825921 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdced3f-a08b-41fb-b59e-0eaf32fe68f1" containerName="payload-pre-processing" Nov 04 01:47:44.826283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.826029 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" Nov 04 01:47:44.826283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.826044 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3775cccb-abcd-441b-8911-828ebe37f451" containerName="maas-api" Nov 04 01:47:44.826283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.826057 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdced3f-a08b-41fb-b59e-0eaf32fe68f1" containerName="payload-pre-processing" Nov 04 01:47:44.826283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.826069 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a5cf1f-7d5e-465a-ba32-08a89016a1c3" containerName="payload-processing" Nov 04 01:47:44.831527 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.831502 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.845321 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.845235 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc"] Nov 04 01:47:44.866686 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.866652 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-7ba35a-serving-cert\"" Nov 04 01:47:44.894373 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.894326 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkv65\" (UniqueName: \"kubernetes.io/projected/50dc8509-4629-49ad-b2e6-510853f32679-kube-api-access-lkv65\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.894554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.894475 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc8509-4629-49ad-b2e6-510853f32679-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.894554 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.894539 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.894662 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.894637 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-metrics-certs\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.963416 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.963385 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr"] Nov 04 01:47:44.963913 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.963899 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="24cf86b4-99fa-46d1-98e5-4ffc4ccc2d84" containerName="istio-proxy" Nov 04 01:47:44.967129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.967114 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:44.976129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.976106 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-7ba35a-dockercfg-sk75v\"" Nov 04 01:47:44.979138 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.979111 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-7ba35a\"" Nov 04 01:47:44.979256 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.979171 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr"] Nov 04 01:47:44.995593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.995566 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkv65\" (UniqueName: \"kubernetes.io/projected/50dc8509-4629-49ad-b2e6-510853f32679-kube-api-access-lkv65\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.995752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.995610 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc8509-4629-49ad-b2e6-510853f32679-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.995752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.995648 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.995752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.995679 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-metrics-certs\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.996320 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.996296 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc8509-4629-49ad-b2e6-510853f32679-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.998115 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.998089 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:44.998210 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:44.998116 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-metrics-certs\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:45.005711 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.005691 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw"] Nov 04 01:47:45.009404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.009388 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.012710 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.012687 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkv65\" (UniqueName: \"kubernetes.io/projected/50dc8509-4629-49ad-b2e6-510853f32679-kube-api-access-lkv65\") pod \"maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:45.022721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.022700 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw"] Nov 04 01:47:45.108492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.108388 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.108492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.108450 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xwq\" (UniqueName: \"kubernetes.io/projected/22163579-cc73-4ebb-9a2e-38748eeedcd9-kube-api-access-82xwq\") pod \"payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.108761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.108508 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22163579-cc73-4ebb-9a2e-38748eeedcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.108761 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.108573 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sw74\" (UniqueName: \"kubernetes.io/projected/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-kube-api-access-6sw74\") pod \"payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.153834 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.153805 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:45.210263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.210181 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.210263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.210248 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xwq\" (UniqueName: \"kubernetes.io/projected/22163579-cc73-4ebb-9a2e-38748eeedcd9-kube-api-access-82xwq\") pod \"payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.210509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.210313 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22163579-cc73-4ebb-9a2e-38748eeedcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.210509 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.210413 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sw74\" (UniqueName: \"kubernetes.io/projected/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-kube-api-access-6sw74\") pod \"payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.211591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.211108 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.211591 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.211505 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22163579-cc73-4ebb-9a2e-38748eeedcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.224225 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.224110 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xwq\" (UniqueName: \"kubernetes.io/projected/22163579-cc73-4ebb-9a2e-38748eeedcd9-kube-api-access-82xwq\") pod \"payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.224756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.224579 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sw74\" (UniqueName: \"kubernetes.io/projected/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-kube-api-access-6sw74\") pod \"payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.283469 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.283435 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.338528 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.338375 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.447520 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.447484 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc"] Nov 04 01:47:45.452373 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:45.452323 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dc8509_4629_49ad_b2e6_510853f32679.slice/crio-ea5e9345f47ad32c6e34f9a9b4d24596481b86727792025e8845ab416feabec8 WatchSource:0}: Error finding container ea5e9345f47ad32c6e34f9a9b4d24596481b86727792025e8845ab416feabec8: Status 404 returned error can't find the container with id ea5e9345f47ad32c6e34f9a9b4d24596481b86727792025e8845ab416feabec8 Nov 04 01:47:45.569384 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:45.569250 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22163579_cc73_4ebb_9a2e_38748eeedcd9.slice/crio-d151cf96f2423c583e192200c367eee5a0cfb97ad48b28be26e1251f415240bc WatchSource:0}: Error finding container d151cf96f2423c583e192200c367eee5a0cfb97ad48b28be26e1251f415240bc: Status 404 returned error can't find the container with id d151cf96f2423c583e192200c367eee5a0cfb97ad48b28be26e1251f415240bc Nov 04 01:47:45.569384 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.569313 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr"] Nov 04 01:47:45.626596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.626569 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw"] Nov 04 01:47:45.627952 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:47:45.627831 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca99aed_75e9_44cf_9fe2_7f90aefc466d.slice/crio-1847212a570ef42ab8d659912fc2dfdd4bef269a7fe8bde38c3bdaa42668713f WatchSource:0}: Error finding container 1847212a570ef42ab8d659912fc2dfdd4bef269a7fe8bde38c3bdaa42668713f: Status 404 returned error can't find the container with id 1847212a570ef42ab8d659912fc2dfdd4bef269a7fe8bde38c3bdaa42668713f Nov 04 01:47:45.980882 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.980855 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" event={"ID":"22163579-cc73-4ebb-9a2e-38748eeedcd9","Type":"ContainerStarted","Data":"2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369"} Nov 04 01:47:45.981379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.980910 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" event={"ID":"22163579-cc73-4ebb-9a2e-38748eeedcd9","Type":"ContainerStarted","Data":"d151cf96f2423c583e192200c367eee5a0cfb97ad48b28be26e1251f415240bc"} Nov 04 01:47:45.981379 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.981058 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:45.982832 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.982803 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" event={"ID":"0ca99aed-75e9-44cf-9fe2-7f90aefc466d","Type":"ContainerStarted","Data":"f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa"} Nov 04 01:47:45.982957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.982849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" event={"ID":"0ca99aed-75e9-44cf-9fe2-7f90aefc466d","Type":"ContainerStarted","Data":"1847212a570ef42ab8d659912fc2dfdd4bef269a7fe8bde38c3bdaa42668713f"} Nov 04 01:47:45.982957 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.982869 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:45.984106 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:45.984081 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" event={"ID":"50dc8509-4629-49ad-b2e6-510853f32679","Type":"ContainerStarted","Data":"ea5e9345f47ad32c6e34f9a9b4d24596481b86727792025e8845ab416feabec8"} Nov 04 01:47:46.004178 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:46.004128 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" podStartSLOduration=2.004112899 podStartE2EDuration="2.004112899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:45.999315196 +0000 UTC m=+2698.022247217" watchObservedRunningTime="2026-11-04 01:47:46.004112899 +0000 UTC m=+2698.027044907" Nov 04 01:47:46.019122 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:46.019085 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" podStartSLOduration=2.019072594 podStartE2EDuration="2.019072594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:46.017013992 +0000 UTC m=+2698.039945997" watchObservedRunningTime="2026-11-04 01:47:46.019072594 +0000 UTC m=+2698.042004601" Nov 04 01:47:46.991975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:46.991933 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" event={"ID":"50dc8509-4629-49ad-b2e6-510853f32679","Type":"ContainerStarted","Data":"e99e588c44eaf0f6055e88a3d87889de97a728a424c4ad9df18b7b0f918f6ab3"} Nov 04 01:47:46.992483 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:46.992361 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:47.012968 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:47.012925 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" podStartSLOduration=2.501870426 podStartE2EDuration="3.012911293s" totalImagesPullingTime="511.040867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:47.011752515 +0000 UTC m=+2699.034684522" watchObservedRunningTime="2026-11-04 01:47:47.012911293 +0000 UTC m=+2699.035843300" Nov 04 01:47:50.966211 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:50.966170 4312 scope.go:122] "RemoveContainer" containerID="3bff1dac4952e35a8a0821c3258e45513821368ccb678c4b4b001a43575cc615" Nov 04 01:47:53.003206 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:53.003179 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:47:56.995485 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:56.995452 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:47:56.995900 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:47:56.995628 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:47:59.459371 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:47:59.459309 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:48:13.023897 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:48:13.023857 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q"] Nov 04 01:48:13.026811 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:48:13.026782 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rdt7q"] Nov 04 01:48:14.131969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:48:14.131933 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30eb8587-68bf-46ee-90a9-d55b3aa61116" path="/var/lib/kubelet/pods/30eb8587-68bf-46ee-90a9-d55b3aa61116/volumes" Nov 04 01:48:29.457144 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:48:29.457063 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:48:30.026830 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:48:30.026790 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv"] Nov 04 01:48:30.029802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:48:30.029768 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-mr8fv"] Nov 04 01:48:30.109022 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:48:30.108988 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0" path="/var/lib/kubelet/pods/86d60fbb-8f8d-416b-bcb9-f9d9aa13ebe0/volumes" Nov 04 01:48:59.457461 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:48:59.457412 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:49:24.459793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:24.459754 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw"] Nov 04 01:49:24.475381 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:24.475319 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr"] Nov 04 01:49:25.293051 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.292914 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc"] Nov 04 01:49:25.293255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.293166 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" podUID="50dc8509-4629-49ad-b2e6-510853f32679" containerName="maas-api" containerID="cri-o://e99e588c44eaf0f6055e88a3d87889de97a728a424c4ad9df18b7b0f918f6ab3" gracePeriod=30 Nov 04 01:49:25.519046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.510946 4312 generic.go:356] "Generic (PLEG): container finished" podID="50dc8509-4629-49ad-b2e6-510853f32679" containerID="e99e588c44eaf0f6055e88a3d87889de97a728a424c4ad9df18b7b0f918f6ab3" exitCode=0 Nov 04 01:49:25.519046 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.511099 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" event={"ID":"50dc8509-4629-49ad-b2e6-510853f32679","Type":"ContainerDied","Data":"e99e588c44eaf0f6055e88a3d87889de97a728a424c4ad9df18b7b0f918f6ab3"} Nov 04 01:49:25.577978 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.577956 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:49:25.623563 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.623534 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-maas-api-tls\") pod \"50dc8509-4629-49ad-b2e6-510853f32679\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " Nov 04 01:49:25.623731 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.623617 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50dc8509-4629-49ad-b2e6-510853f32679-kube-api-access-lkv65\" (UniqueName: \"kubernetes.io/projected/50dc8509-4629-49ad-b2e6-510853f32679-kube-api-access-lkv65\") pod \"50dc8509-4629-49ad-b2e6-510853f32679\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " Nov 04 01:49:25.623794 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.623760 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50dc8509-4629-49ad-b2e6-510853f32679-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc8509-4629-49ad-b2e6-510853f32679-openshift-service-ca\") pod \"50dc8509-4629-49ad-b2e6-510853f32679\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " Nov 04 01:49:25.623866 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.623848 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-metrics-certs\") pod \"50dc8509-4629-49ad-b2e6-510853f32679\" (UID: \"50dc8509-4629-49ad-b2e6-510853f32679\") " Nov 04 01:49:25.624255 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.624161 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dc8509-4629-49ad-b2e6-510853f32679-openshift-service-ca" pod "50dc8509-4629-49ad-b2e6-510853f32679" (UID: "50dc8509-4629-49ad-b2e6-510853f32679"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:25.624442 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.624366 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc8509-4629-49ad-b2e6-510853f32679-openshift-service-ca\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:25.626119 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.626091 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-metrics-certs" pod "50dc8509-4629-49ad-b2e6-510853f32679" (UID: "50dc8509-4629-49ad-b2e6-510853f32679"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:49:25.626281 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.626176 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-maas-api-tls" pod "50dc8509-4629-49ad-b2e6-510853f32679" (UID: "50dc8509-4629-49ad-b2e6-510853f32679"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:49:25.626375 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.626361 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dc8509-4629-49ad-b2e6-510853f32679-kube-api-access-lkv65" pod "50dc8509-4629-49ad-b2e6-510853f32679" (UID: "50dc8509-4629-49ad-b2e6-510853f32679"). InnerVolumeSpecName "kube-api-access-lkv65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:25.725175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.725139 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkv65\" (UniqueName: \"kubernetes.io/projected/50dc8509-4629-49ad-b2e6-510853f32679-kube-api-access-lkv65\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:25.725175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.725167 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-metrics-certs\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:25.725175 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:25.725176 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dc8509-4629-49ad-b2e6-510853f32679-maas-api-tls\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:26.517521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:26.517443 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" Nov 04 01:49:26.517521 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:26.517452 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc" event={"ID":"50dc8509-4629-49ad-b2e6-510853f32679","Type":"ContainerDied","Data":"ea5e9345f47ad32c6e34f9a9b4d24596481b86727792025e8845ab416feabec8"} Nov 04 01:49:26.517719 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:26.517538 4312 scope.go:122] "RemoveContainer" containerID="e99e588c44eaf0f6055e88a3d87889de97a728a424c4ad9df18b7b0f918f6ab3" Nov 04 01:49:26.536801 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:26.536765 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc"] Nov 04 01:49:26.543071 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:26.543042 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ba35a-75c8fc6fbf-j6mmc"] Nov 04 01:49:28.107607 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:28.107571 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50dc8509-4629-49ad-b2e6-510853f32679" path="/var/lib/kubelet/pods/50dc8509-4629-49ad-b2e6-510853f32679/volumes" Nov 04 01:49:29.460461 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.460413 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" podUID="0ca99aed-75e9-44cf-9fe2-7f90aefc466d" containerName="payload-processing" containerID="cri-o://f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa" gracePeriod=25 Nov 04 01:49:29.460974 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:49:29.460552 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:49:29.475966 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.475932 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" podUID="22163579-cc73-4ebb-9a2e-38748eeedcd9" containerName="payload-pre-processing" containerID="cri-o://2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369" gracePeriod=25 Nov 04 01:49:29.743802 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.743767 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:49:29.765970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.765942 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:49:29.865606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.865549 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-kube-api-access-6sw74\" (UniqueName: \"kubernetes.io/projected/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-kube-api-access-6sw74\") pod \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " Nov 04 01:49:29.865793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.865653 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22163579-cc73-4ebb-9a2e-38748eeedcd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22163579-cc73-4ebb-9a2e-38748eeedcd9-plugins-config-volume\") pod \"22163579-cc73-4ebb-9a2e-38748eeedcd9\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " Nov 04 01:49:29.865793 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.865732 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22163579-cc73-4ebb-9a2e-38748eeedcd9-kube-api-access-82xwq\" (UniqueName: \"kubernetes.io/projected/22163579-cc73-4ebb-9a2e-38748eeedcd9-kube-api-access-82xwq\") pod \"22163579-cc73-4ebb-9a2e-38748eeedcd9\" (UID: \"22163579-cc73-4ebb-9a2e-38748eeedcd9\") " Nov 04 01:49:29.865879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.865804 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-plugins-config-volume\") pod \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\" (UID: \"0ca99aed-75e9-44cf-9fe2-7f90aefc466d\") " Nov 04 01:49:29.866174 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.866144 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22163579-cc73-4ebb-9a2e-38748eeedcd9-plugins-config-volume" pod "22163579-cc73-4ebb-9a2e-38748eeedcd9" (UID: "22163579-cc73-4ebb-9a2e-38748eeedcd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:29.866303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.866148 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-plugins-config-volume" pod "0ca99aed-75e9-44cf-9fe2-7f90aefc466d" (UID: "0ca99aed-75e9-44cf-9fe2-7f90aefc466d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:29.867737 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.867716 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-kube-api-access-6sw74" pod "0ca99aed-75e9-44cf-9fe2-7f90aefc466d" (UID: "0ca99aed-75e9-44cf-9fe2-7f90aefc466d"). InnerVolumeSpecName "kube-api-access-6sw74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:29.867820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.867784 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22163579-cc73-4ebb-9a2e-38748eeedcd9-kube-api-access-82xwq" pod "22163579-cc73-4ebb-9a2e-38748eeedcd9" (UID: "22163579-cc73-4ebb-9a2e-38748eeedcd9"). InnerVolumeSpecName "kube-api-access-82xwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:29.967695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.967644 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xwq\" (UniqueName: \"kubernetes.io/projected/22163579-cc73-4ebb-9a2e-38748eeedcd9-kube-api-access-82xwq\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:29.967695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.967686 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:29.967695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.967696 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sw74\" (UniqueName: \"kubernetes.io/projected/0ca99aed-75e9-44cf-9fe2-7f90aefc466d-kube-api-access-6sw74\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:29.967695 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:29.967707 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22163579-cc73-4ebb-9a2e-38748eeedcd9-plugins-config-volume\") on node \"ip-10-0-136-96.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:30.541525 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.541486 4312 generic.go:356] "Generic (PLEG): container finished" podID="22163579-cc73-4ebb-9a2e-38748eeedcd9" containerID="2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369" exitCode=0 Nov 04 01:49:30.542000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.541552 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" Nov 04 01:49:30.542000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.541578 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" event={"ID":"22163579-cc73-4ebb-9a2e-38748eeedcd9","Type":"ContainerDied","Data":"2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369"} Nov 04 01:49:30.542000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.541660 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr" event={"ID":"22163579-cc73-4ebb-9a2e-38748eeedcd9","Type":"ContainerDied","Data":"d151cf96f2423c583e192200c367eee5a0cfb97ad48b28be26e1251f415240bc"} Nov 04 01:49:30.542000 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.541692 4312 scope.go:122] "RemoveContainer" containerID="2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369" Nov 04 01:49:30.542974 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.542900 4312 generic.go:356] "Generic (PLEG): container finished" podID="0ca99aed-75e9-44cf-9fe2-7f90aefc466d" containerID="f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa" exitCode=0 Nov 04 01:49:30.543042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.542974 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" Nov 04 01:49:30.543042 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.542987 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" event={"ID":"0ca99aed-75e9-44cf-9fe2-7f90aefc466d","Type":"ContainerDied","Data":"f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa"} Nov 04 01:49:30.543125 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.543040 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw" event={"ID":"0ca99aed-75e9-44cf-9fe2-7f90aefc466d","Type":"ContainerDied","Data":"1847212a570ef42ab8d659912fc2dfdd4bef269a7fe8bde38c3bdaa42668713f"} Nov 04 01:49:30.550996 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.550978 4312 scope.go:122] "RemoveContainer" containerID="2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369" Nov 04 01:49:30.551272 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:49:30.551245 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369\": container with ID starting with 2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369 not found: ID does not exist" containerID="2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369" Nov 04 01:49:30.551393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.551271 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369"} err="failed to get container status \"2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369\": rpc error: code = NotFound desc = could not find container \"2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369\": container with ID starting with 2e147bfba982d85b7fe34ba47eba3bd673bc9d9625eb99621510b8e0d5f9f369 not found: ID does not exist" Nov 04 01:49:30.551393 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.551287 4312 scope.go:122] "RemoveContainer" containerID="f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa" Nov 04 01:49:30.560069 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.560016 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw"] Nov 04 01:49:30.560141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.560094 4312 scope.go:122] "RemoveContainer" containerID="f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa" Nov 04 01:49:30.560413 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:49:30.560395 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa\": container with ID starting with f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa not found: ID does not exist" containerID="f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa" Nov 04 01:49:30.560467 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.560419 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa"} err="failed to get container status \"f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa\": rpc error: code = NotFound desc = could not find container \"f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa\": container with ID starting with f8d1cf22150be857e0b730f39c1b7d16a07ab762affec45450065ac01bf9bcaa not found: ID does not exist" Nov 04 01:49:30.564181 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.564160 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ba35a-585986fb8b-7qppw"] Nov 04 01:49:30.577589 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.577568 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr"] Nov 04 01:49:30.584059 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:30.584038 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ba35a-7bc49647b7-ts7pr"] Nov 04 01:49:32.108975 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:32.108936 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca99aed-75e9-44cf-9fe2-7f90aefc466d" path="/var/lib/kubelet/pods/0ca99aed-75e9-44cf-9fe2-7f90aefc466d/volumes" Nov 04 01:49:32.109441 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:32.109350 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22163579-cc73-4ebb-9a2e-38748eeedcd9" path="/var/lib/kubelet/pods/22163579-cc73-4ebb-9a2e-38748eeedcd9/volumes" Nov 04 01:49:51.015823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:51.015792 4312 scope.go:122] "RemoveContainer" containerID="4b52ff98dcef80f6d402ef22f65b5f30c7a0298dbbf12d18c761cf151ef7dc22" Nov 04 01:49:51.024366 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:49:51.024348 4312 scope.go:122] "RemoveContainer" containerID="d5201de6a9f5ea28089d9023e1096560b32e4e2b04cf84ec98e3a1aafffd065e" Nov 04 01:49:59.667613 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:49:59.667518 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:50:29.457137 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:50:29.457099 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:50:34.539196 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:34.539160 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-568986c44-m79cq_64ec9c7b-6735-43a7-a859-e7186474c121/manager/0.log" Nov 04 01:50:35.425732 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:35.425695 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-vmh4f_0a14699c-70df-4f2c-850d-ee52d69d89de/manager/0.log" Nov 04 01:50:35.616864 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:35.616830 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cgt26_b1513474-cbe8-4a4b-a267-9230887fe173/server/0.log" Nov 04 01:50:37.557905 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:37.557854 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bd788d5f-cwfvx_4feca7d9-7167-4353-b078-cf7f9278df85/maas-api/0.log" Nov 04 01:50:38.126542 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:38.126514 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-qr9rp_786f49ba-6bb9-441d-9c8a-8f2d5e8f3299/revoke-keys/0.log" Nov 04 01:50:38.297076 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:38.297042 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-2g4kn_023a276e-0c6e-4b23-9d79-25c5bfc8af3b/revoke-keys/0.log" Nov 04 01:50:38.653103 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:38.653072 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6e15d7d5-n9rrh_837ccc7d-89bb-4a35-bdfa-f015487b2664/revoke-keys/0.log" Nov 04 01:50:41.418219 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:41.418192 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sdxks_12f17e0b-b49a-4e4b-b4a3-f65d75989758/kuadrant-console-plugin/0.log" Nov 04 01:50:41.638284 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:41.638245 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-246jq_40914eb2-0c22-498c-b294-04f0fc9daf6c/manager/0.log" Nov 04 01:50:44.009493 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:44.009462 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-pzcnv_d6f638e8-b3b0-48dc-bb30-55f37633f3f7/kube-auth-proxy/0.log" Nov 04 01:50:44.201317 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:44.201286 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-4l6sk_33e3cbdc-f101-49ff-953e-c46a13d2658e/istio-proxy/0.log" Nov 04 01:50:45.124886 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:45.124854 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-kvnxf_2ee19db5-0a09-4b15-b268-e749a0a2969f/istio-proxy/0.log" Nov 04 01:50:45.316251 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:45.316219 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-m8lgv_d73cdf89-4461-4224-8cf1-53353a26afd5/istio-proxy/0.log" Nov 04 01:50:45.675831 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:45.675803 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-ng8g5_a085702c-fd6e-46d0-b282-1f96cf558e24/istio-proxy/0.log" Nov 04 01:50:46.221944 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:46.221913 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-4xmvn_a0d81c57-ee60-4be3-b9b2-e0d10514088c/payload-pre-processing/0.log" Nov 04 01:50:46.391004 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:46.390973 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:50:47.739756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:47.739724 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9_a7f1bc94-14c3-4783-9aad-2efc85a264e1/main/0.log" Nov 04 01:50:47.917910 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:47.917871 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k42tl_870c4be8-527c-43ad-a63f-39dd02718c55/main/0.log" Nov 04 01:50:48.107148 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:48.107114 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ghv26_61ca9501-204d-4daf-a56b-0794b0f8f6d0/main/0.log" Nov 04 01:50:48.479569 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:48.479487 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk_ba71124b-dc83-408b-8e5c-1b9abd6fac9a/main/0.log" Nov 04 01:50:48.846675 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:48.846641 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8_44760365-91dc-4a74-b176-231e25a8df60/main/0.log" Nov 04 01:50:49.025941 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:49.025905 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz_f659cf1c-b69f-436b-8bf0-9bbddd61383d/main/0.log" Nov 04 01:50:59.415502 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:50:59.415472 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-568986c44-m79cq_64ec9c7b-6735-43a7-a859-e7186474c121/manager/0.log" Nov 04 01:50:59.456198 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:50:59.456166 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:51:00.272585 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:00.272550 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-vmh4f_0a14699c-70df-4f2c-850d-ee52d69d89de/manager/0.log" Nov 04 01:51:00.508146 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:00.508120 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cgt26_b1513474-cbe8-4a4b-a267-9230887fe173/server/0.log" Nov 04 01:51:01.461303 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:01.461272 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bd788d5f-cwfvx_4feca7d9-7167-4353-b078-cf7f9278df85/maas-api/0.log" Nov 04 01:51:02.059430 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:02.059400 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-qr9rp_786f49ba-6bb9-441d-9c8a-8f2d5e8f3299/revoke-keys/0.log" Nov 04 01:51:02.250411 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:02.250385 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-2g4kn_023a276e-0c6e-4b23-9d79-25c5bfc8af3b/revoke-keys/0.log" Nov 04 01:51:02.639273 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:02.639240 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6e15d7d5-n9rrh_837ccc7d-89bb-4a35-bdfa-f015487b2664/revoke-keys/0.log" Nov 04 01:51:04.769067 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:04.769038 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sdxks_12f17e0b-b49a-4e4b-b4a3-f65d75989758/kuadrant-console-plugin/0.log" Nov 04 01:51:04.955668 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:04.955632 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-246jq_40914eb2-0c22-498c-b294-04f0fc9daf6c/manager/0.log" Nov 04 01:51:06.497557 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:06.497528 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-pzcnv_d6f638e8-b3b0-48dc-bb30-55f37633f3f7/kube-auth-proxy/0.log" Nov 04 01:51:06.695226 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:06.695190 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-4l6sk_33e3cbdc-f101-49ff-953e-c46a13d2658e/istio-proxy/0.log" Nov 04 01:51:07.652258 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:07.652230 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-kvnxf_2ee19db5-0a09-4b15-b268-e749a0a2969f/istio-proxy/0.log" Nov 04 01:51:07.876447 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:07.876415 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-m8lgv_d73cdf89-4461-4224-8cf1-53353a26afd5/istio-proxy/0.log" Nov 04 01:51:08.267596 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:08.267566 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-ng8g5_a085702c-fd6e-46d0-b282-1f96cf558e24/istio-proxy/0.log" Nov 04 01:51:08.830746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:08.830718 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-4xmvn_a0d81c57-ee60-4be3-b9b2-e0d10514088c/payload-pre-processing/0.log" Nov 04 01:51:09.022535 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:09.022504 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:51:09.763559 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:09.763530 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-vbvc9_a7f1bc94-14c3-4783-9aad-2efc85a264e1/main/0.log" Nov 04 01:51:09.960849 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:09.960821 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k42tl_870c4be8-527c-43ad-a63f-39dd02718c55/main/0.log" Nov 04 01:51:10.146746 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:10.146711 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ghv26_61ca9501-204d-4daf-a56b-0794b0f8f6d0/main/0.log" Nov 04 01:51:10.518421 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:10.518325 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xgsvk_ba71124b-dc83-408b-8e5c-1b9abd6fac9a/main/0.log" Nov 04 01:51:10.908899 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:10.908865 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-f7bp8_44760365-91dc-4a74-b176-231e25a8df60/main/0.log" Nov 04 01:51:11.098473 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:11.098440 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rgsrz_f659cf1c-b69f-436b-8bf0-9bbddd61383d/main/0.log" Nov 04 01:51:20.024749 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:20.024717 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn"] Nov 04 01:51:20.026918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:20.026895 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2g4kn"] Nov 04 01:51:20.107714 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:20.107682 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023a276e-0c6e-4b23-9d79-25c5bfc8af3b" path="/var/lib/kubelet/pods/023a276e-0c6e-4b23-9d79-25c5bfc8af3b/volumes" Nov 04 01:51:29.457139 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:51:29.457053 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:51:34.024410 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:34.024374 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp"] Nov 04 01:51:34.027404 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:34.027379 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qr9rp"] Nov 04 01:51:34.107420 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:34.107387 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786f49ba-6bb9-441d-9c8a-8f2d5e8f3299" path="/var/lib/kubelet/pods/786f49ba-6bb9-441d-9c8a-8f2d5e8f3299/volumes" Nov 04 01:51:45.093348 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:45.093298 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-vmh4f_0a14699c-70df-4f2c-850d-ee52d69d89de/manager/0.log" Nov 04 01:51:45.335904 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:45.335866 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-vmh4f_0a14699c-70df-4f2c-850d-ee52d69d89de/manager/0.log" Nov 04 01:51:46.013973 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:46.013942 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:51:46.183001 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:46.182964 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:51:52.023628 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:52.023598 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh"] Nov 04 01:51:52.026921 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:52.026894 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e15d7d5-n9rrh"] Nov 04 01:51:52.106593 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:51:52.106565 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837ccc7d-89bb-4a35-bdfa-f015487b2664" path="/var/lib/kubelet/pods/837ccc7d-89bb-4a35-bdfa-f015487b2664/volumes" Nov 04 01:51:59.456534 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:51:59.456496 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:52:23.409868 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:23.409830 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-vmh4f_0a14699c-70df-4f2c-850d-ee52d69d89de/manager/0.log" Nov 04 01:52:23.676177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:23.676092 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-vmh4f_0a14699c-70df-4f2c-850d-ee52d69d89de/manager/0.log" Nov 04 01:52:24.389911 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.389884 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:52:24.558682 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.558648 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:52:24.866751 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.866649 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lsdgx/must-gather-fpkmp"] Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867104 4312 cpu_manager.go:414] "containerMap: removing container" podUID="22163579-cc73-4ebb-9a2e-38748eeedcd9" containerName="payload-pre-processing" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867116 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22163579-cc73-4ebb-9a2e-38748eeedcd9" containerName="payload-pre-processing" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867129 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0ca99aed-75e9-44cf-9fe2-7f90aefc466d" containerName="payload-processing" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867134 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca99aed-75e9-44cf-9fe2-7f90aefc466d" containerName="payload-processing" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867145 4312 cpu_manager.go:414] "containerMap: removing container" podUID="50dc8509-4629-49ad-b2e6-510853f32679" containerName="maas-api" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867150 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dc8509-4629-49ad-b2e6-510853f32679" containerName="maas-api" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867221 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="22163579-cc73-4ebb-9a2e-38748eeedcd9" containerName="payload-pre-processing" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867229 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca99aed-75e9-44cf-9fe2-7f90aefc466d" containerName="payload-processing" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.867236 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="50dc8509-4629-49ad-b2e6-510853f32679" containerName="maas-api" Nov 04 01:52:24.870709 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.870695 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:24.874729 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.874712 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsdgx\"/\"openshift-service-ca.crt\"" Nov 04 01:52:24.875987 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.875966 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsdgx\"/\"kube-root-ca.crt\"" Nov 04 01:52:24.876109 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.875988 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lsdgx\"/\"default-dockercfg-d5q5k\"" Nov 04 01:52:24.888750 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.888729 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsdgx/must-gather-fpkmp"] Nov 04 01:52:24.998478 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.998447 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8688\" (UniqueName: \"kubernetes.io/projected/99e5c273-8edc-418f-83bf-a8f588f40458-kube-api-access-s8688\") pod \"must-gather-fpkmp\" (UID: \"99e5c273-8edc-418f-83bf-a8f588f40458\") " pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:24.998637 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:24.998482 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99e5c273-8edc-418f-83bf-a8f588f40458-must-gather-output\") pod \"must-gather-fpkmp\" (UID: \"99e5c273-8edc-418f-83bf-a8f588f40458\") " pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:25.099282 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.099244 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8688\" (UniqueName: \"kubernetes.io/projected/99e5c273-8edc-418f-83bf-a8f588f40458-kube-api-access-s8688\") pod \"must-gather-fpkmp\" (UID: \"99e5c273-8edc-418f-83bf-a8f588f40458\") " pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:25.099491 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.099305 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99e5c273-8edc-418f-83bf-a8f588f40458-must-gather-output\") pod \"must-gather-fpkmp\" (UID: \"99e5c273-8edc-418f-83bf-a8f588f40458\") " pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:25.099687 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.099668 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99e5c273-8edc-418f-83bf-a8f588f40458-must-gather-output\") pod \"must-gather-fpkmp\" (UID: \"99e5c273-8edc-418f-83bf-a8f588f40458\") " pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:25.111820 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.111794 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8688\" (UniqueName: \"kubernetes.io/projected/99e5c273-8edc-418f-83bf-a8f588f40458-kube-api-access-s8688\") pod \"must-gather-fpkmp\" (UID: \"99e5c273-8edc-418f-83bf-a8f588f40458\") " pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:25.184371 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.184290 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsdgx/must-gather-fpkmp" Nov 04 01:52:25.456752 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.456715 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsdgx/must-gather-fpkmp"] Nov 04 01:52:25.459204 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:52:25.459179 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99e5c273_8edc_418f_83bf_a8f588f40458.slice/crio-df5a7432556ba73cbd8085182cf14c24f391e6702ade7651fae2783f17bdbe1f WatchSource:0}: Error finding container df5a7432556ba73cbd8085182cf14c24f391e6702ade7651fae2783f17bdbe1f: Status 404 returned error can't find the container with id df5a7432556ba73cbd8085182cf14c24f391e6702ade7651fae2783f17bdbe1f Nov 04 01:52:25.460969 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.460950 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:52:25.560391 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:25.560360 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsdgx/must-gather-fpkmp" event={"ID":"99e5c273-8edc-418f-83bf-a8f588f40458","Type":"ContainerStarted","Data":"df5a7432556ba73cbd8085182cf14c24f391e6702ade7651fae2783f17bdbe1f"} Nov 04 01:52:26.572634 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:26.572589 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsdgx/must-gather-fpkmp" event={"ID":"99e5c273-8edc-418f-83bf-a8f588f40458","Type":"ContainerStarted","Data":"410128cc94b8e4aacfee26abfb0f73059dd2d834296b5bb700b051a6c178d22f"} Nov 04 01:52:27.583551 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:27.583515 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsdgx/must-gather-fpkmp" event={"ID":"99e5c273-8edc-418f-83bf-a8f588f40458","Type":"ContainerStarted","Data":"2c7aa34b7b9a2181e34f2fbee1ad93636e39562238acc025ba2d6c6b9871bdc6"} Nov 04 01:52:27.603581 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:27.603521 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lsdgx/must-gather-fpkmp" podStartSLOduration=2.6339613440000003 podStartE2EDuration="3.603508082s" totalImagesPullingTime="969.546738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:52:27.601836325 +0000 UTC m=+2979.624768332" watchObservedRunningTime="2026-11-04 01:52:27.603508082 +0000 UTC m=+2979.626440089" Nov 04 01:52:28.153692 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:28.153661 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h7zqj_389e7a15-7659-450c-9483-d59ea607e94d/global-pull-secret-syncer/0.log" Nov 04 01:52:28.436513 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:28.436440 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cs7zr_dc46f559-06ba-4445-8048-f29d73b9f168/konnectivity-agent/0.log" Nov 04 01:52:28.530715 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:28.530684 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-96.us-east-2.compute.internal_916a1141c4b97a1387808b2945274410/haproxy/0.log" Nov 04 01:52:29.530141 ip-10-0-136-96 kubenswrapper[4312]: E1104 01:52:29.530069 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d6x9n" containerName="extractor" Nov 04 01:52:35.275612 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:35.275527 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sdxks_12f17e0b-b49a-4e4b-b4a3-f65d75989758/kuadrant-console-plugin/0.log" Nov 04 01:52:35.378357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:35.372900 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-246jq_40914eb2-0c22-498c-b294-04f0fc9daf6c/manager/0.log" Nov 04 01:52:36.900357 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:36.900309 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_90c035f1-3ad5-4ce4-acf6-a84cf6ec433c/alertmanager/0.log" Nov 04 01:52:36.928077 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:36.928044 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_90c035f1-3ad5-4ce4-acf6-a84cf6ec433c/config-reloader/0.log" Nov 04 01:52:36.958756 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:36.958725 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_90c035f1-3ad5-4ce4-acf6-a84cf6ec433c/kube-rbac-proxy-web/0.log" Nov 04 01:52:36.984154 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:36.984126 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_90c035f1-3ad5-4ce4-acf6-a84cf6ec433c/kube-rbac-proxy/0.log" Nov 04 01:52:37.038096 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.038056 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_90c035f1-3ad5-4ce4-acf6-a84cf6ec433c/kube-rbac-proxy-metric/0.log" Nov 04 01:52:37.063906 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.063876 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_90c035f1-3ad5-4ce4-acf6-a84cf6ec433c/prom-label-proxy/0.log" Nov 04 01:52:37.091721 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.091683 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_90c035f1-3ad5-4ce4-acf6-a84cf6ec433c/init-config-reloader/0.log" Nov 04 01:52:37.504026 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.502361 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-8nrdl_facd3002-3a6e-4511-bd92-ac51951ba75f/monitoring-plugin/0.log" Nov 04 01:52:37.707353 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.706846 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xr28v_c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf/node-exporter/0.log" Nov 04 01:52:37.733665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.733635 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xr28v_c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf/kube-rbac-proxy/0.log" Nov 04 01:52:37.764028 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.763956 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xr28v_c8b2fc9f-84d5-4e6d-8acb-362c8ba7b6bf/init-textfile/0.log" Nov 04 01:52:37.797531 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.797485 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wm6w4_f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a/kube-rbac-proxy-main/0.log" Nov 04 01:52:37.821198 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.821156 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wm6w4_f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a/kube-rbac-proxy-self/0.log" Nov 04 01:52:37.845841 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:37.845813 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wm6w4_f2c1d676-2ee8-4722-b0c1-fa605ee6ec0a/openshift-state-metrics/0.log" Nov 04 01:52:38.172606 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.172572 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-hkx52_c2c762ba-d7cd-42e2-8410-1bf6fe2408a3/prometheus-operator-admission-webhook/0.log" Nov 04 01:52:38.279648 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.279604 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq"] Nov 04 01:52:38.287032 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.287002 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.303141 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.303110 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq"] Nov 04 01:52:38.326177 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.326131 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9cd5b47-fcwvh_da8a4c19-339b-45a6-9502-b02e0bc17482/thanos-query/0.log" Nov 04 01:52:38.370918 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.370867 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9cd5b47-fcwvh_da8a4c19-339b-45a6-9502-b02e0bc17482/kube-rbac-proxy-web/0.log" Nov 04 01:52:38.388248 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.388217 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-proc\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.388458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.388264 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-podres\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.388458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.388374 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-sys\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.388458 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.388422 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-lib-modules\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.388653 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.388620 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5k8k\" (UniqueName: \"kubernetes.io/projected/83995c05-f4d8-45bf-ab66-1e77d0041760-kube-api-access-r5k8k\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.399437 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.399408 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9cd5b47-fcwvh_da8a4c19-339b-45a6-9502-b02e0bc17482/kube-rbac-proxy/0.log" Nov 04 01:52:38.426663 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.426593 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9cd5b47-fcwvh_da8a4c19-339b-45a6-9502-b02e0bc17482/prom-label-proxy/0.log" Nov 04 01:52:38.453427 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.453384 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9cd5b47-fcwvh_da8a4c19-339b-45a6-9502-b02e0bc17482/kube-rbac-proxy-rules/0.log" Nov 04 01:52:38.484153 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.484106 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9cd5b47-fcwvh_da8a4c19-339b-45a6-9502-b02e0bc17482/kube-rbac-proxy-metrics/0.log" Nov 04 01:52:38.489743 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.489713 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k8k\" (UniqueName: \"kubernetes.io/projected/83995c05-f4d8-45bf-ab66-1e77d0041760-kube-api-access-r5k8k\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.489879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.489791 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-proc\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.489879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.489839 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-podres\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.489879 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.489874 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-sys\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.490056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.489912 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-lib-modules\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.490056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.489912 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-proc\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.490056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.489959 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-podres\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.490056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.490000 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-sys\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.490056 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.490038 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83995c05-f4d8-45bf-ab66-1e77d0041760-lib-modules\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.502399 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.502371 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5k8k\" (UniqueName: \"kubernetes.io/projected/83995c05-f4d8-45bf-ab66-1e77d0041760-kube-api-access-r5k8k\") pod \"perf-node-gather-daemonset-wfqrq\" (UID: \"83995c05-f4d8-45bf-ab66-1e77d0041760\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.614550 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.614512 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:38.912263 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:38.912223 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq"] Nov 04 01:52:38.916447 ip-10-0-136-96 kubenswrapper[4312]: W1104 01:52:38.916404 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod83995c05_f4d8_45bf_ab66_1e77d0041760.slice/crio-9f50a3d0e0c06c572d709f6a8795f3f98ce8796ab4e1cf88053b21e4d8063286 WatchSource:0}: Error finding container 9f50a3d0e0c06c572d709f6a8795f3f98ce8796ab4e1cf88053b21e4d8063286: Status 404 returned error can't find the container with id 9f50a3d0e0c06c572d709f6a8795f3f98ce8796ab4e1cf88053b21e4d8063286 Nov 04 01:52:39.716665 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:39.716626 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" event={"ID":"83995c05-f4d8-45bf-ab66-1e77d0041760","Type":"ContainerStarted","Data":"0b3d6c9fad8bc53985bf8a248ffbf5b238c0ab7df16c3160e5b7dfc1c8909702"} Nov 04 01:52:39.717269 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:39.717240 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" event={"ID":"83995c05-f4d8-45bf-ab66-1e77d0041760","Type":"ContainerStarted","Data":"9f50a3d0e0c06c572d709f6a8795f3f98ce8796ab4e1cf88053b21e4d8063286"} Nov 04 01:52:39.717492 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:39.717468 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:39.742202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:39.741606 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" podStartSLOduration=1.741587201 podStartE2EDuration="1.741587201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:52:39.739178203 +0000 UTC m=+2991.762110211" watchObservedRunningTime="2026-11-04 01:52:39.741587201 +0000 UTC m=+2991.764519209" Nov 04 01:52:40.869688 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:40.869648 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-gvtrm_0e4ff47b-fe50-4203-bf00-1d332e02ff6d/download-server/0.log" Nov 04 01:52:42.619200 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:42.619090 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f9sn6_3f5cd56a-808e-4b32-a685-127a21c6cf3d/dns/0.log" Nov 04 01:52:42.643167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:42.643139 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f9sn6_3f5cd56a-808e-4b32-a685-127a21c6cf3d/kube-rbac-proxy/0.log" Nov 04 01:52:42.838172 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:42.838132 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9cpl7_b58d7d01-e446-4636-8aa6-55302737e478/dns-node-resolver/0.log" Nov 04 01:52:43.462601 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:43.462525 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74fd556577-xc96m_705fa7e2-620e-4425-8d01-8f1335e2b13c/registry/0.log" Nov 04 01:52:43.545166 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:43.545139 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pzkfb_4c61d96e-9340-453b-854a-944a058cac37/node-ca/0.log" Nov 04 01:52:45.544316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:45.544287 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-pzcnv_d6f638e8-b3b0-48dc-bb30-55f37633f3f7/kube-auth-proxy/0.log" Nov 04 01:52:45.581249 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:45.581194 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-4l6sk_33e3cbdc-f101-49ff-953e-c46a13d2658e/istio-proxy/0.log" Nov 04 01:52:45.743202 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:45.743161 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-kvnxf_2ee19db5-0a09-4b15-b268-e749a0a2969f/istio-proxy/0.log" Nov 04 01:52:45.776283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:45.776252 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-m8lgv_d73cdf89-4461-4224-8cf1-53353a26afd5/istio-proxy/0.log" Nov 04 01:52:45.844167 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:45.844094 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-ng8g5_a085702c-fd6e-46d0-b282-1f96cf558e24/istio-proxy/0.log" Nov 04 01:52:45.937088 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:45.937059 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-4xmvn_a0d81c57-ee60-4be3-b9b2-e0d10514088c/payload-pre-processing/0.log" Nov 04 01:52:45.965129 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:45.965088 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:52:46.741700 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:46.741671 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-wfqrq" Nov 04 01:52:48.136874 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:48.136829 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8wrmb_bbcde58b-e496-453a-9ec1-18007d6bf8b8/serve-healthcheck-canary/0.log" Nov 04 01:52:48.940823 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:48.940786 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d6x9n_b97127b8-d6e3-47f1-bcab-63a53d06c574/kube-rbac-proxy/0.log" Nov 04 01:52:48.969305 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:48.969273 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d6x9n_b97127b8-d6e3-47f1-bcab-63a53d06c574/exporter/0.log" Nov 04 01:52:48.999621 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:48.999580 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d6x9n_b97127b8-d6e3-47f1-bcab-63a53d06c574/extractor/0.log" Nov 04 01:52:50.303773 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:50.303739 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-pzcnv_d6f638e8-b3b0-48dc-bb30-55f37633f3f7/kube-auth-proxy/0.log" Nov 04 01:52:50.328617 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:50.328584 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-4l6sk_33e3cbdc-f101-49ff-953e-c46a13d2658e/istio-proxy/0.log" Nov 04 01:52:50.421607 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:50.421579 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-kvnxf_2ee19db5-0a09-4b15-b268-e749a0a2969f/istio-proxy/0.log" Nov 04 01:52:50.438574 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:50.438518 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-m8lgv_d73cdf89-4461-4224-8cf1-53353a26afd5/istio-proxy/0.log" Nov 04 01:52:50.482876 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:50.482843 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-ng8g5_a085702c-fd6e-46d0-b282-1f96cf558e24/istio-proxy/0.log" Nov 04 01:52:50.533730 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:50.533694 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-4xmvn_a0d81c57-ee60-4be3-b9b2-e0d10514088c/payload-pre-processing/0.log" Nov 04 01:52:50.558152 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:50.558065 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log" Nov 04 01:52:51.074387 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:51.074356 4312 scope.go:122] "RemoveContainer" containerID="e2ea5e54ef3096cf750dede550e2f3632e074f47ce993f68c4cc1051c6650325" Nov 04 01:52:51.091055 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:51.090711 4312 scope.go:122] "RemoveContainer" containerID="ded606ab9e04dea32a3f00b525e7de7728c828350b66ef7ce91735254f9b8282" Nov 04 01:52:51.113917 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:51.112431 4312 scope.go:122] "RemoveContainer" containerID="1b67081abfd559368c84543f37807e82acc4453f83ef4166504f40aa11510267" Nov 04 01:52:52.074927 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:52.074895 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8ff7fd4ff-56hwl_c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6/manager/0.log" Nov 04 01:52:52.094576 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:52.094545 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8ff7fd4ff-56hwl_c19dc4b5-9a3d-4e5a-ab01-52ef8293a5b6/copy-manifests/0.log" Nov 04 01:52:54.078388 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:54.078350 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-568986c44-m79cq_64ec9c7b-6735-43a7-a859-e7186474c121/manager/0.log" Nov 04 01:52:54.868304 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:54.868272 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-vmh4f_0a14699c-70df-4f2c-850d-ee52d69d89de/manager/0.log" Nov 04 01:52:54.899970 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:54.899940 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cgt26_b1513474-cbe8-4a4b-a267-9230887fe173/server/0.log" Nov 04 01:52:54.962283 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:54.962249 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-pzcnv_d6f638e8-b3b0-48dc-bb30-55f37633f3f7/kube-auth-proxy/0.log" Nov 04 01:52:54.986539 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:54.986513 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-4l6sk_33e3cbdc-f101-49ff-953e-c46a13d2658e/istio-proxy/0.log" Nov 04 01:52:55.090745 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:55.090712 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-kvnxf_2ee19db5-0a09-4b15-b268-e749a0a2969f/istio-proxy/0.log" Nov 04 01:52:55.115452 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:55.115423 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-m8lgv_d73cdf89-4461-4224-8cf1-53353a26afd5/istio-proxy/0.log" Nov 04 01:52:55.161954 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:55.161868 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-ng8g5_a085702c-fd6e-46d0-b282-1f96cf558e24/istio-proxy/0.log" Nov 04 01:52:55.224316 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:55.224284 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-4xmvn_a0d81c57-ee60-4be3-b9b2-e0d10514088c/payload-pre-processing/0.log" Nov 04 01:52:55.241716 ip-10-0-136-96 kubenswrapper[4312]: I1104 01:52:55.241683 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-lggqw_5d475784-6dfc-4d23-8e67-70c7cca43636/payload-processing/0.log"